diff --git a/Archive/LipstickOnAPigExceptionHierarchy.md b/Archive/LipstickOnAPigExceptionHierarchy.md index add559f1f..0603f4077 100644 --- a/Archive/LipstickOnAPigExceptionHierarchy.md +++ b/Archive/LipstickOnAPigExceptionHierarchy.md @@ -1,9 +1,9 @@ -# Lipstick on a Pig Exception Hierarchy Proposal +# Lipstick on a Pig Exception Hierarchy Proposal -## Overview / Purpose +## Overview / Purpose -The [wiki:Archive/LipstickOnAPig Lipstick on a Pig] strike force is attempting to create a standard exception hierarchy for repy to provide a unified method to allow users to handle exceptions within repy programs. +The [Lipstick on a Pig](LipstickOnAPig.md) strike force is attempting to create a standard exception hierarchy for repy to provide a unified method to allow users to handle exceptions within repy programs. Currently the Repy library has inconsistent treatment and use of exceptions, which can be confusing and off-putting for new users. Our goal is to come up with a clear, uniform and logical way to present exceptions. The repy library functions have specific needs and goals, and the reflection hierarchy we're proposing reflects this. We are focusing on creating the core exception hierarchy which will be used by the repy library calls themselves; exceptions specific to any seattlelib code will need to augment the hierarchy with their own exceptions. @@ -15,9 +15,9 @@ The hierarchy we designed is meant to account for the specific types of errors w ---- - -## Proposed Repy Hierarchy + +## Proposed Repy Hierarchy ---- Below is our proposed exception hierarchy. The RepyException tree is separate from Exception to show how the two are meant to be distinct, although RepyException is a subclass of Python's Exception. @@ -53,21 +53,21 @@ PythonException +-- Builtin exceptions raised due to programmer mistakes (AttributeError, IndexError, NameError, etc) ``` - -## Comparison to Python and Current Repy Implementation + +## Comparison to Python and Current Repy Implementation ---- The python builtins tend to throw the same types of errors, and repy is meant to mimic this behavior. For example, builtins overwhelmingly raise ```TypeError``` when passed an argument of the wrong type, and raise ```ValueError``` when passed an incorrect value (such as a positive number instead of a negative one). Repy exceptions are also meant to be less cryptic than some of the python exceptions. The socket.* errors for example were quite cryptic for new users, which resulted in the creation of ```AddressError``` rather than using python's ```socket.gaierror```. - -### Reused Python Exceptions + +### Reused Python Exceptions ---- -#### TypeError +#### TypeError The python library uses [TypeError](http://docs.python.org/library/exceptions.html#exceptions.TypeError) to notify users that an argument of the wrong type was passed to a function. We want to emulate this behavior, as the current repy implementation primarily raises ```Exception``` and gives the description of what argument was incorrect. For example, the python ```abs``` built-in expects a numeric type. If a string is passed in, it raises a ```TypeError```. @@ -76,7 +76,7 @@ Similar behaviour is proposed for repy. For example, if a user tries specifying -#### ValueError +#### ValueError We want to change the Repy Library to emulate python's builtin API's usage of [ValueError](http://docs.python.org/library/exceptions.html#exceptions.ValueError). Although exact Python behavior differs per implementation and per call (a bad thing, in our opinion) ```ValueError``` tends to be used to indicate that a value of the correct type was passed in, but an invalid value was used. An example of this is the ```sleep``` API call, which expects a numeric value greater than 0. If a negative number is passed in, a ValueError should be raised @@ -105,9 +105,9 @@ socket.gethostbyname_ex("") ``` While this could be considered a network error, it seems reasonable to assume that getting host information from the empty string is obviously a value error, no network checks have to be performed. The repy version of the above code should raise a ```ValueError``` - -### Network Examples + +### Network Examples ---- Network failures can be of particular importance given the nature of repy and the types of programs that tend to get written in it. There are a large number of reasons for why network operations can fail, and we feel python's socket module's interface could be improved upon. @@ -116,7 +116,7 @@ Native python's exception types for network related errors are pretty limited. T Our goal is to make it easy for users to determine precisely what went wrong with a network call, and be able to recover gracefully from these error conditions. -#### Host Name Example +#### Host Name Example In native python, socket errors are defined in the socket module. Since repy has no concept of modules, and doesn't allow imports, it doesn't make sense to use the code snippet below. ```python import socket @@ -137,7 +137,7 @@ except NetworkAddressError: #recover from error ``` -#### NetworkAddressError +#### NetworkAddressError One class of errors that users may tend to see often due to the nature of resolution is address errors. In Python these are presented as ```socket.gaierror```, and what the specific cause of the problem is presented in the text of the exception. Currently several repy network API calls still make use of ```socket.*``` errors. However the socket module is not in the scope of repy user programs, and modules aren't allowed to be imported. This means the current implementation of repy requires workarounds: @@ -167,7 +167,7 @@ except NetworkError: #handle error condition ``` -#### Connection Example +#### Connection Example The way of interacting with sockets is also different from regular python: ```python @@ -200,7 +200,7 @@ finally: The exception handing for both cases are meant to be similar. We aimed to keep the programming model familiar enough to programmers with a python background, but also more user friendly to those just staring out with repy. Just as ```openconn``` and ```sendmess``` provide easy access for users to interact over networks, ```NetworkError``` and its subclasses are meant to facilitate convenient handling of common errors that can occur. -#### ConnectionRefusedError +#### ConnectionRefusedError In python, the following code snippet raises a ```socket.error```: ```python import socket @@ -217,7 +217,7 @@ s = openconn('127.0.0.1', 12347, '127.0.0.1', 12345) #a ConnectionRefusedError would be raised ``` -#### NetworkDisconnectedError +#### NetworkDisconnectedError This error is meant to be raised by ```getmyip()```, to indicate that a computer lost it's connection to the internet. The current behavior is ambiguous; ```getmyip``` raises a ```socket.gairerror``` with the message No address associated with hostname". This error should also be raised if the socket layer raises ```socket.error``` 101: "Network is unreachable". This is currently the exception that is thrown by various network API calls, such as when ```sendmess``` is given an external ip but the network is disconnected @@ -225,16 +225,16 @@ This error should also be raised if the socket layer raises ```socket.error``` 1 - -### File System Examples + +### File System Examples ---- While the python standard library provides many functions to flexibly interface with the underlying file system, the nature of repy requires a much more restricted subset of allowed operations. Additionally, there are a number of areas (such as file system modes) where the python standard library functions fail "silently" by either not raising an exception or proceeding with unexpected behavior. The goal of the repy file system exceptions are to make it very clear what error has occurred and to not silently continue. -#### File Open Examples +#### File Open Examples In native python, most errors with the file system raise a general "IOError" with more specific details in the message. This makes it hard to realize what happened: @@ -257,7 +257,7 @@ except FileNotFoundError: pass ``` -#### File Open Mode Example +#### File Open Mode Example Python "silently" allows some obscure modes with the file system: @@ -278,7 +278,7 @@ except InvalidFileModeException: pass ``` -#### Illegal File Names +#### Illegal File Names Python allows many characters within the open method such as ".." to traverse directories. Because Repy has no concept of "directories" for programs running in the VMs, it must be more restrictive of allowed file names. @@ -299,9 +299,9 @@ except IllegalFileNameException: pass ``` - -### Resource Errors + +### Resource Errors ---- Given the nature of repy, it is necessary to be able to signal users that they have exceeded their allowed resources. There are several types of resources, and some of these errors may be recoverable, some may not. For example, when a program is consuming too much memory, there is no way for a program to catch that exception. However, if a user tries to have too many files open or have too many event handlers at once, it will be possible to catch the exception raised during the attempt to create more, and allow the programmer to take appropriate action. @@ -310,9 +310,9 @@ Recoverable resource errors generally fall into two categories: attempting to us When trying to use a resource that wasn't permitted in a VMl's restrictions file, a ResourceNotAllowedError will be raised. In the case of overusing a resource, a ResourceExhaustedError will be raised. These will make it more clear why an error occured when trying to use a node's resources, and avoid the hacky string parsing currently required for recovery. - -## Internal Errors + +## Internal Errors ---- The above errors are meant to be the exceptions exposed to users. If a corrupt internal state is detected by repy, an ```InternalError``` will be raised. When this occurs, a user's repy program will terminate instantly. The rationale behind this is that we want to hide potential repy weaknesses to typical users, and ensure that a corrupt VM doesn't resume execution in a potentially harmful state. @@ -324,12 +324,12 @@ If this occurs in a ```try...finally``` block, the finally block will not get ex Information regarding the internal error will get privately logged, to provide seattle developers some insight regarding what caused the problem, and do postmortem analysis. - -## Unresolved Questions/Issues + +## Unresolved Questions/Issues ---- There are a number of problems/questions we weren't sure the most appropriate way to handle. We'd appreciate any feedback regarding these issues * What type of error should be raised if a user's IP address changes? If a program has event handlers associated with an IP address and that changes, how should the user be notified? * The biggest concern for changing IP's is how to notify the user. It's easy to imagine the scenario where the only thing keeping a program in execution (rather than terminating) is event handlers waiting to be triggered. So if a program has a recvmess handler waiting to receive UDP packets, but the machine's IP changes, how do we notify the user in a meaningful way? If there's no context to which allow them to catch an exception we raised, does it even make sense to? Plus, recvmess is provided a localip, which is no longer valid! It's conceivable that users would write their own logic to periodically check the IP, and if it changes, then to update the event handlers accordingly. As to how/if repy should detect these (and possibly unregister stale listeners) is still uncertain... - * Should an exception should be thrown if the interface and destination IP don't match? I.e. ```sendmess("google.com",80, "hi", "127.0.0.1", 12345)``` \ No newline at end of file + * Should an exception should be thrown if the interface and destination IP don't match? I.e. ```sendmess("google.com",80, "hi", "127.0.0.1", 12345)``` diff --git a/Contributing/BuildInstructions.md b/Contributing/BuildInstructions.md index 041a87959..3c072889b 100644 --- a/Contributing/BuildInstructions.md +++ b/Contributing/BuildInstructions.md @@ -69,6 +69,13 @@ directory which will be created as a sub-directory inside your main repository. It takes its instructions from `config_initialize.txt` to fetch the dependencies. +Make sure that you are using python 2.7 for initializing and building to +avoid any errors with building. To use python 2.7, you can do: + +```sh +$ python2 initialize.py +``` + 3. To build a runnable component from the source dependencies, run the `scripts/build.py` script. You may supply it an optional target folder for the build (which must be created first). Name this folder as you like. diff --git a/Contributing/ContributorsPage.md b/Contributing/ContributorsPage.md index b667f9bfe..3dcc9ba33 100644 --- a/Contributing/ContributorsPage.md +++ b/Contributing/ContributorsPage.md @@ -1,41 +1,42 @@ -# Contributor Page +# Contributor Page This page contains information useful to developers working on Seattle. Anything that is essential for each developer to read is **in bold text**. Note: We've recently moved our VCS and ticket system to GitHub. Please see our [organization's page](https://github.com/SeattleTestbed) for the latest code, to discuss open issues, and to create forks off our code base for your contributions! -**If you are a new contributor, please look at our [wiki:GettingStarted Getting Started] page.** +**If you are a new contributor, please look at our [Getting Started](README.md) page.** * Accessing our GitHub repositories, building components, Github & Git tutorial, and using the wiki * Check out (i.e., ```git clone```) our source code from [https://github.com/SeattleTestbed] - * [wiki:BuildInstructions Instructions] for building Seattle components + * [Instructions](BuildInstructions.md) for building Seattle components - * [wiki:Local/RepoAccess Github and Git] instructions + * [Github and Git](../Archive/Local/RepoAccess.md) instructions * [Useful Git Commands](http://zackperdue.com/tutorials/super-useful-need-to-know-git-commands) * [How to resolve a merge conflict with Git](https://help.github.com/articles/resolving-a-merge-conflict-from-the-command-line) - * [wiki:SubmittingAPatch How to submit a patch for inclusion if you do not have commit access] - * ~~**[wiki:ContributorAccounts Getting accounts] for Seattle Wiki, testbed machines, and seattle-devel**~~ - * ~~**[wiki:ManagingTracTickets Managing Trac tickets]**~~ + * [How to submit a patch for inclusion if you do not have commit access](SubmittingAPatch.md) * Programming in Repy * The [Python Tutorial](http://docs.python.org/tutorial/) from the Python site. - * [wiki:PythonTutorial Subset of Python Tutorial] with basic information about the subset of Python needed to write code in Repy - * [wiki:PythonVsRepy Python You Need to Forget to Use Repy] - * **[wiki:RepyTutorial Repy Tutorial]** - * [wiki:RepyApi Repy Library] reference - * [wiki:SeattleLib Seattle Standard Library] reference - * [wiki:PortingPythonToRepy Porting Python to Repy Guide] - * [wiki:RepyHelper Importing Repy code into Python using repyhelper] - * [wiki:RepyV1vsRepyV2 Difference between Repy V1 and Repy V2] + * [PythonTutorial Subset of Python Tutorial](../Programming/PythonTutorial.md) with basic information about the subset of Python needed to write code in Repy + * [Python Vs Repy Python You Need to Forget to Use Repy](../Programming/PythonVsRepy.md) + * [Python vs RePyV2 Python You need to Forget to Use RePyV2](../Programming/PythonVsRepyV2.md) + * **[Repy Tutorial](../Programming/RepyTutorial.md)** + * **[RePyV2 Tutorial](../Programming/RepyV2Tutorial.md)** + * [Repy Library](../Programming/RepyApi.md) reference + * [RepyV2 Library](../Programming/RepyV2Api.md) reference + * [Seattle Standard Library](../Programming/SeattleLib_v1/SeattleLib.md) reference + * [Porting Python to Repy Guide](../Programming/PortingPythonToRepy.md) + * [Importing Repy code into Python using repyhelper](../Programming/RepyHelper.md) + * [Difference between Repy V1 and Repy V2](../Programming/RepyV1vsRepyV2.md) * Testing - * **[wiki:UnitTestFramework Writing Tests for Seattle's Unit Test Framework]** + * **[Writing Tests for Seattle's Unit Test Framework](UnitTestFramework.md)** * **[wiki:UnitTests Running Repy and / or node manager unit tests]** - * [wiki:UnitTestFrameworkRunning Supplemental information about Running Tests with Seattle's Unit Test Framework] - * [wiki:UpdaterUnitTests Running Software Updater Unit Tests] - * [wiki:IntegrationTestFramework Writing Integration Tests] - * [wiki:Local/RunningIntegrationTests Running Integration Tests] - * [wiki:ClearinghouseInstallation Running a Clearinghouse test server] - * [wiki:Archive/CustomInstallerBuilderTesting Running a Custom Installer Builder test server] + * [Supplemental information about Running Tests with Seattle's Unit Test Framework](UnitTestFrameworkRunning.md) + * [Running Software Updater Unit Tests](../Outdated/UpdaterUnitTests.md) + * [Writing Integration Tests](../Operating/IntegrationTestFramework.md) + * [Running Integration Tests](../Archive/Local/RunningIntegrationTests.md) + * [Running a Clearinghouse test server](../Operating/Clearinghouse/Installation.md) + * [Running a Custom Installer Builder test server](../Archive/CustominstallerBuilderTesting.md) * [wiki:RemoteTestingService Running remote tests on seattle testbeds] * **Continuous Integration via Travis-CI and AppVeyor:** * [For team members - managing CI](https://github.com/SeattleTestbed/seattlelib_v2/wiki/Continuous-Integration-for-the-Team) @@ -43,11 +44,11 @@ Note: We've recently moved our VCS and ticket system to GitHub. Please see our [ * Style guidelines for documentation, python code, web code * **[Python/Repy style guidelines](https://github.com/secure-systems-lab/code-style-guidelines) for most Seattle programming** - * [wiki:WebCodingStyle Web style guidelines] for web programming - * **[wiki:Local/WikiFormatting Documentation format guidelines] for wiki pages** + * [Web style guidelines](WebCodingStyle.md) for web programming + * **[Documentation format guidelines](../Archive/Local/WikiFormatting.md) for wiki pages** * Installation instructions and documentation - * [wiki:InstallerDocumentation Formal Installer Documentation] + * [Formal Installer Documentation](../UnderstandingSeattle/InstallerDocumentation.md) * [wiki:SeattleDownload Download and installation instructions] for Seattle * [wiki:ClearinghouseInstallation Installing your own instance of Seattle Clearinghouse] * [wiki:CustomInstallerBuilder Using the Custom Installer Builder] @@ -58,34 +59,31 @@ Note: We've recently moved our VCS and ticket system to GitHub. Please see our [ * [wiki:CustomInstallerBuilderApi Custom Installer Builder XML-RPC API] * Deployment - * [wiki:BaseInstallers Building] the base installers + * [Building](../Operating/BaseInstallers.md) the base installers * [wiki:NsisSystemSetup System setup instructions] to be able to build the Windows GUI installer - * [wiki:Local/VersionDeployment Deploying] a new version - * [wiki:SoftwareUpdaterSetup Setting up] the software updater - * [wiki:Archive/SeattleGeniProductionHttp Updating] the production Seattle Clearinghouse code - * [wiki:BuildDemokit Building the Demokit] - * [wiki:Libraries/Overlord Overlord Deployment and Monitoring Library] for deploying persistent services on VMs + * [Deploying](../Archive/Local/VersionDeployment.md.md) a new version + * [Setting up](../Operating/SoftwareUpdaterSetup.md) the software updater + * [Updating](../Archive/SeattleGeniProductionHttp.md) the production Seattle Clearinghouse code + * [Building the Demokit](../Operating/BuildDemokit.md) + * [Overlord Deployment and Monitoring Library](../Archive/Libraries/Overlord.md) for deploying persistent services on VMs * Developer Resources - * [wiki:Libraries/ExperimentLibrary Experiment Library] for use in scripting communication with nodes, etc. + * [Experiment Library](../Archive/Libraries/ExperimentLibrary.md) for use in scripting communication with nodes, etc. * [wiki:DevelopingWithEclipse Developing with Eclipse] - * ~~Project Resources~~ - * ~~[wiki:SeattleResources A full listing of Seattle project services]~~ - * ~~[wiki:Local/ContributorAccountManagement Creating and deleting accounts for developers]~~ + * Project Resources * [wiki:SeattleAdminResources Resources needed by Seattle sysadmin] * Events relevant to Seattle and Seattle team information - * [wiki:SeattleTalks Upcoming talks] about Seattle - * **[wiki:Local/ContributorContactInfo Project Members] contact information** + * [Upcoming talks](../SeattleTalks.md) about Seattle + * **[Project Members](../Archive/Local/ContributorContactInfo.md) contact information** * Other - * [wiki:SeattleBackend Seattle Backend] - * [wiki:ContainmentInSeattle] about controlling node outgoing traffic - * ~~[wiki:Archive/MobileCeNotes Windows CE] implementation notes and problems~~ - * [wiki:BenchmarkCustomInstallerInfo Benchmark and Custom Installer Info] implementation notes and problems. - * [wiki:RepyNetworkRestrictions Repy Network Restrictions] information and errata. - * [wiki:Libraries/StatisticsLibrary Statistics Library] for analyzing date regarding nodes & VMs on Seattle. - * [wiki:Applications/GeoIpServer Running a GeoIP Server] - * How to run the [wiki:RunningSecLayerBenchmarks security layer benchmarks] - * [SeleXor](http://selexor.poly.edu/) for additional control over the VM acquisition process. \ No newline at end of file + * [Seattle Backend](../UnderstandingSeattle/SeattleShellBackend.md) + * [Containment In Seattle](../Outdated/ContainmentInSeattle.md) about controlling node outgoing traffic + * [Benchmark and Custom Installer Info](../UnderstandingSeattle/BenchmarkCustomInstallerInfo.md) implementation notes and problems. + * [Repy Network Restrictions](../Programming/RepyNetworkRestrictions.md) information and errata. + * [Statistics Library](../Archive/Libraries/StatisticsLibrary.md) for analyzing date regarding nodes & VMs on Seattle. + * [Running a GeoIP Server](../Applications/GeoIpServer.md) + * How to run the [Security Layer benchmarks](../OutdatedRunningSecLayerBenchmarks.md) + * [SeleXor](http://selexor.poly.edu/) for additional control over the VM acquisition process. diff --git a/Contributing/README.md b/Contributing/README.md index 79453454b..3e555c4ef 100644 --- a/Contributing/README.md +++ b/Contributing/README.md @@ -10,7 +10,7 @@ the following: 1. Start by reading the [Seattle docs](../README.md). This will give you a better sense of what Seattle is and how its structured, and contains pointers - to the programming documentation as well. The [Contributor's Page](ContributorsPage.wiki) + to the programming documentation as well. The [Contributor's Page](ContributorsPage.md) has even more reading material and links. 2. Join our [mailing list](https://groups.google.com/forum/?hl=en#!forum/seattle-devel)! @@ -24,7 +24,7 @@ has even more reading material and links. insights that you may have, and also post any concerns or questions that arise. 4. When you are done with your issue, make sure your code adheres to our - [code style guidelines](../Programming/CodingStyle.wiki). Have someone else on + [code style guidelines](WebCodingStyle.md). Have someone else on the team review your code. When they give you the OK, send a pull request. If your code is good, someone will merge in your changes. diff --git a/EducationalAssignments/ABStoragePartOne.md b/EducationalAssignments/ABStoragePartOne.md index 1da473ec7..cd14e56f3 100644 --- a/EducationalAssignments/ABStoragePartOne.md +++ b/EducationalAssignments/ABStoragePartOne.md @@ -7,8 +7,8 @@ concept that refers to an abstract machine that mediates all access to objects by subjects. This can be used to allow, deny, or change the behavior of any set of calls. While not a perfect way of validating your reference monitor, it is useful to create test cases to see whether your -security layer will work as expected. (The test cases may be turned in as -part of the next assignment.) +security layer will work as expected (the test cases may be turned in as +part of the next assignment). Please ask your instructor if test cases are available to you, some instructors may provide with test cases. This assignment is intended to reinforce concepts about access control and reference monitors in a hands-on manner. @@ -16,47 +16,51 @@ reference monitors in a hands-on manner. - - ## Overview ---- In this assignment you will create a security layer which keeps a backup copy of a file in case it is written incorrectly. This is a common technique for things like firmware images where a system may not be able to -recover if the file is written incorrectly. For this assignment, every -`correct' file must start with the character 'S' and end with the character +recover if the file is written incorrectly. For this assignment, a +valid file must start with the character 'S' and end with the character 'E'. If any other characters (including lowercase 's', 'e', etc.) are the first or last characters, then the file is considered invalid. -However, you must permit the application to write information into the file. -The application should not be blocked from performing any writeat() operation, -because when it chooses it may later write 'S' at the start and 'E' at the -end. Note that checking if the file starts with 'S' and ends with 'E' is -only performed when close is called. - -You may store two copies of A/B files on disk, one that is the valid backup -(which is used for reading) and the other that is written to. When an -app calls ABopenfile(), this indicates that the A/B files, which you should -name filename.a and filename.b, should be opened. -When the app calls readat(), all reads must be performed on the valid -file. Similarly, when the app calls writeat(), all writes must be -performed on the invalid file. If the app uses ABopenfile() to create a -file that does not exist (by setting create=True when calling ABopenfile()), -the reference monitor will create a new file 'SE' in filename.a and an empty -file called filename.b. When close() is called on the file, if a file is -not valid, it is discarded. if both files are valid, the older one is -discarded. - -Note that the behavior of other file system calls should remain unchanged. -This means listfiles(), removefile(), and calls to files accessed with -openfile() instead of ABopenfile() remain unchanged by this reference monitor. +Applications use ABopenfile() to create or open a file. Files are created by +setting create=True. When calling ABopenfile(), the reference +monitor will create a valid backup file called filename.a and an empty +file we will write to called filename.b. When close() is called on the file, if both filename.a and filename.b are valid, the original file's data is replaced with the data of filename.b. If filename.b is not valid, the original file should use the data of the backup filename.a file. Afterward both the filename.a and filename.b file should be deleted, and only the original file should remain. + +Write test applications to ensure your reference monitor behaves properly +in different cases and to test attacks against your monitor. + +#### The Reference Monitor Must: +1. Not modify or disable any functionality of any [RepyV2 API calls](../Programming/RepyV2API.md), such as: + * Creating new files + * Opening an existing file + * Reading valid backup using readat() + * Writing to file using writeat(). This includes invalid writes, because 'S' and 'E' +may later be written to the begining and end of the file respectively. +2. Check if file contents starts with 'S' and ends with 'E', only when close() is called. +3. Update the original file with the new data IF the new data is valid on close(). +4. Not produce any errors + * Normal operations should not be blocked or produce any output + * Invalid operations should not produce any output to the user +#### The Reference Monitor Should: +1. Create two copies of the same file (filename.a and filename.b) + * One is a valid backup to read from, and the other is written to +2. When an app calls ABopenfile(), the method opens the A/B files, which + you should name filename.a and filename.b. +3. When the app calls readat(), all reads must be performed on the valid backup. +4. When the app calls writeat(), all writes must be performed on the written to file. + Three design paradigms are at work in this assignment: accuracy, efficiency, and security. * Accuracy: The security layer should only stop certain actions from being blocked. All other actions should be allowed. For example, if an app -tries to read data from a valid file, this must succeed as per normal and +tries to read data from the backup file, this must succeed as per normal and must not be blocked. All situations that are not described above *must* match that of the underlying API. @@ -65,8 +69,8 @@ so performance is not compromised. For example, keeping a complete copy of every file on disk in memory would be forbidden. * Security: The attacker should not be able to circumvent the security -layer. Hence, if the attacker can cause an invalid file to be read or can -write to a valid file, then the security is compromised, for example. +layer. For example, if the attacker can cause an invalid file to be saved, read the "write to" file, or can +write to the backup file we read from, then the security is compromised. @@ -76,12 +80,12 @@ Please refer to the [SeattleTestbed Build Instructions](../Contributing/BuildIns for details. Once you have built RepyV2 into a directory of your choice, change into that -directory. Use the command below in order to run your RepyV2 programs: +directory. Use the command below in order to run your RepyV2 applications: -```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [program].r2py``` +```python2 repy.py restrictions.default encasementlib.r2py [security_layer].r2py [application].r2py``` -(Replace `[security_layer].r2py` and `[program].r2py` by the names of the -security layers and program that you want to run.) +(Replace '[security_layer].r2py' and '[application].r2py' by the names of the +security layers and application that you want to run.) In order to test whether or not these steps worked, please copy and paste the code found below for the sample security layer and sample attack. @@ -110,7 +114,7 @@ to run repy files. + + +### Tutorials for Repy and Python +---- +Now that you have Repy and Python, you may need a refresher on how to use +them. The following tutorials provide this information. + + * Official [Python tutorial](http://docs.python.org/tutorial/) + * [Differences between RepyV2 and Python](../Programming/PythonVsRepyV2.md) + * List of [RepyV2 API calls](../Programming/RepyV2API.md) + + + +## Building the security layer +---- +The following program is a sample security layer, it is not complete and does not +handle all cases required by the API. Remember, you have no idea how the +attacker will try to penetrate your security layer, so it is important that +you leave nothing to chance! + + +### A basic (and inadequate) defense + +Time to start coding! Let's inspect a basic security layer. + +``` +""" +This security layer inadequately handles LeftPad writeat()s that strictly append + + + +Note: + This security layer uses encasementlib.r2py, restrictions.default, repy.py and Python + Also you need to give it an application to run. + python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py + + """ +TYPE="type" +ARGS="args" +RETURN="return" +EXCP="exceptions" +TARGET="target" +FUNC="func" +OBJC="objc" + +class LPFile(): + def __init__(self,filename,create): + # globals + mycontext['debug'] = False + self.LPfile = openfile(filename,create) + self.length = 0 + + def readat(self, bytes, offset): + # Read from the file using the sandbox's readat... + return self.LPfile.readat(bytes, offset) + + def writeat(self,data,offset): + if not offset == self.length: + # write the data and update the length (BUG?) + self.LPfile.writeat(data,offset) + self.length = offset + len(data) + + else: + + if '\n' not in data: + self.LPfile.writeat(data,offset) + else: # bug? + loc = data.find('\n') + # bug? + self.LPfile.writeat(data[:loc]+" "+data[loc:],offset) + + + def close(self): + self.LPfile.close() + + +def LPopenfile(filename, create): + return LPFile(filename,create) + + + + +# The code here sets up type checking and variable hiding for you. You +# should not need to change anything below here. +sec_file_def = {"obj-type":LPFile, + "name":"LPFile", + "writeat":{"type":"func","args":(str,(int,long)),"exceptions":Exception,"return":(int,type(None)),"target":LPFile.writeat}, + "readat":{"type":"func","args":((int,long,type(None)),(int,long)),"exceptions":Exception,"return":str,"target":LPFile.readat}, + "close":{"type":"func","args":None,"exceptions":Exception,"return":(bool,type(None)),"target":LPFile.close} + } + +CHILD_CONTEXT_DEF["openfile"] = {TYPE:OBJC,ARGS:(str,bool),EXCP:Exception,RETURN:sec_file_def,TARGET:LPopenfile} + +# Execute the user code +secure_dispatch_module() +``` + + + +### Testing your security layer +---- +In this part of the assignment you will pretend to be an attacker. Remember +the attacker's objective is to bypass the A/B restrictions or cause +the security layer to act in a disallowed manner. By understanding how the +attacker thinks, you will be able to write better security layers. + +An example of an attack is found below: + +``` +# clean up if the file exists. +if "testfile.txt" in listfiles(): + removefile("testfile.txt") + +myfile=openfile("testfile.txt",True) #Create a file + +myfile.writeat("12345678",0) # no difference, no '\n' + +myfile.writeat("Hi!",0) # writing early in the file + +myfile.writeat("Append!\nShould be indented!!!",8) # strictly appending... + +assert(' ' == myfile.readat(1,17)) # this location should contain a space... + +#Close the file +myfile.close() + + +``` + +If the reference monitor is correct, there should be no assertion failure... + +**Note:** All attacks should be written as Repy V2 files, using the .r2py extension. + +#### Choice of File Names +---- +Filenames may only be in the current directory and may only contain lowercase letters, numbers, the hyphen, underscore, and period characters. Also, filenames cannot be '.', '..', the blank string or start with a period. There is no concept of a directory or a folder in repy. Filenames must be no more than 120 characters long. + +### Running your security layer +---- +Finally, type the following commands at the terminal to run your security +layer with your attack program + +```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py ``` + +Make sure you went through the "How to get RepyV2" section! + + +# Notes and Resources +---- + + * For a complete list of syntax in Repyv2 please visit: + * **[https://github.com/SeattleTestbed/docs/blob/master/Programming/RepyV2API.md]** + + * The following link is an excellent source for information about security layers: **[https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf]** + + * **Note:** It is possible to add multiple security layers to Repy, this +may be useful for testing different mitigations separately. This is +done with the following command at the terminal: + +```python repy.py restrictions.default encasementlib.r2py [security_layer1].r2py [security_layer2].r2py [security_layer3].r2py [program].r2py``` + +**Your security layer must produce no output!! ** + + * In repy log replaces print from python. This may be helpful when +testing if Repy installed correctly. + + +# What to turn in? +---- + * Turn in a repy file called reference_monitor_[ netid ].r2py with all +letters in lowercase. + +* **Never raise unexpected errors or produce any output.** Your program +must produce no output when run normally. diff --git a/EducationalAssignments/LeftPadPartThree.md b/EducationalAssignments/LeftPadPartThree.md new file mode 100644 index 000000000..1361c0a05 --- /dev/null +++ b/EducationalAssignments/LeftPadPartThree.md @@ -0,0 +1,66 @@ + +# Fixing your security layer + +In this assignment you will analyze the bugs in your security layer from [Part One](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/LeftPadPartOne.md) and fix them. You may want to use test cases from [Part Two](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/LeftPadPartTwo.md) to help identify these bugs. + + + +## Overview +---- +In this assignment you are fixing your reference monitor. You have been sent a bunch of test programs that try to compromise your reference monitor. Your job will be to determine where your reference monitor failed, and fix it to ensure an attacker cannot circumvent the security layer. + + + +## Common Bugs + * The reference monitor needs to track the state of the information on disk, but cannot re-read it for every access (due to efficiency concerns). A common mistake is when the attacker can cause the reference monitor’s state to diverge from the underlying system’s state, especially in error conditions. + + * Time-of-check-to-time-of-use (TOCTTOU) bugs and other types of race conditions are a fairly common oversight for students. Some students write test cases that attempt to trigger a race condition to exploit these problems. This can result in essentially any sort of attack, even infinite loops in the reference monitor in some cases. + + * Some reference monitors inappropriately share state for different files. For example, there may be a global set of state that is used to store the first two bytes. By opening multiple files, an attacker may be able to overwrite this state and cause security and accuracy issues. + + * In rare cases, a student’s reference monitor may inappropriately retain state for a file. For example, an attacker may create a file, write some data, then close and delete the file. If the attacker recreates a file with the same name, the state should be cleared. + + * Many reference monitors had accuracy or security bugs as a result of not properly following the instructions. + + + + +## Example : Sample Implementation + +``` +class LPFile(): + def __init__(self,filename,create): + # globals + mycontext['debug'] = False + self.LPfile = openfile(filename,create) + self.length = 0 + + def readat(self, bytes, offset): + # Read from the file using the sandbox's readat... + return self.LPfile.readat(bytes, offset) + + def writeat(self,data,offset): + if not offset == self.length: + # write the data and update the length (BUG?) + self.LPfile.writeat(data,offset) + self.length = offset + len(data) + + else: + + if '\n' not in data: + self.LPfile.writeat(data,offset) + else: # bug? + loc = data.find('\n') + # bug? + self.LPfile.writeat(data[:loc]+" "+data[loc:],offset) +``` + +### Code Analysis +Let's analyze one of the bugs in this implementation. According to the specifications in [Part One](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/LeftPadPartOne.md), it should return RepyArgumentError if there are two '\n' which isn't implemented in this code, moreover, this code inserts spaces before '\n', as a result you can't see the indentation. Another bug is that it should first update the length and write. Furthermore, this code doesn't check if len(data) is greater than self.length when offset is less than self.length. It also assumes self.length = 0 which means that the file would be empty. Moreover, This example doesn't use any locks so you can introduce race conditions and perform a variety of attacks. + + +## What to turn in? +---- + * Never raise unexpected errors or produce any output. Your program must produce no output when run normally. + * Turn in the reference monitor that you have fixed. The name of the reference monitor should be in the form of reference_monitor_[netId].r2py +All letters must be lowercase. diff --git a/EducationalAssignments/LeftPadPartTwo.md b/EducationalAssignments/LeftPadPartTwo.md new file mode 100644 index 000000000..d8166ada5 --- /dev/null +++ b/EducationalAssignments/LeftPadPartTwo.md @@ -0,0 +1,133 @@ + +# Security layer testing and penetration + +In this assignment you will learn how to attack a reference monitor. The reference monitor you will be testing uses the security layer framework (encasement library, etc.) for the Seattle testbed. It is possible to do this assignment separately, but it is recommended that this assignment be completed after [Part One](LeftPadPartOne.md). Either way you should already have a working security layer or access to one. Testing the security layer is done by running a series of test cases that an adversary may use to circumvent your system. This assignment is intended to prepare you for thinking about security paradigms in a functional way. The ideas of information, security and privacy have been embedded into the steps of this assignment. + + +## Overview +---- +In this assignment you are a tester. You have been sent a bunch of reference monitors that need testing before they are deployed. Your job will be to ensure an attacker cannot circumvent these security layers. In order to do this, you will attempt to write, and append invalid data. If you are able to do so, then the security layer is not secure. The future of the system depends on your ability to test code thoroughly! + +Three design paradigms are at work in this assignment: accuracy, efficiency, and security. + + * Accuracy: The security layer should only stop certain actions from being blocked. All other actions should be allowed. + + * Efficiency: The security layer should use a minimum number of resources, so performance is not compromised. + + * Security: The attacker should not be able to circumvent the security layer. + +Within the context of this assignment these design paradigms translate to: + + * Accuracy: The security layer should only modify certain operations (a strictly appending writeat() with one '\n') and raise an exception for certain other actions (a strictly appending writeat() with more than one '\n'). All situations that are not described above must match that of the underlying API. + + * Efficiency: The security layer should use a minimum number of resources, so performance is not compromised. For example, you may not call readat() everytime writeat() is called. It is permissible to call readat() upon fileopen(), however. + + * Security: The attacker should not be able to circumvent the security layer. For example, if the attacker can cause a non-strictly appending write to have ' ' inserted after '\n' or can cause the reference monitor to incorrectly error or hang, then the security is compromised. + +You will submit a zip file containing all of the tests you have created. You will gain points for every student's reference monitor you find a flaw in. It is good if multiple tests of yours break a student's reference monitor, but you gain the same number of tests whether one or more tests break the layer. + + +## Prerequisites + +This assignment assumes you have both the latest Python 2.7 and RepyV2 +installed on your computer. Please refer to the [SeattleTestbed Build Instructions](../Contributing/BuildInstructions.md#prerequisites) +for information on how to get them. + + +### Helpful links +---- +The following links will aid students in becoming comfortable with Python, Repy and seattle: + * Official [Python tutorial](http://docs.python.org/tutorial/) + * [Differences between RepyV2 and Python](../Programming/PythonVsRepyV2.md) + * List of [RepyV2 API calls](../Programming/RepyV2API.md) + + +## Testing security layers +---- +### Hypothesis, test case, counter example + +The goal of a good tester is to test hypotheses. A hypothesis is just a scientific way of asking a question. The hypothesis of this assignment is "This security layer is well designed." The questions you will ask when running your test cases will always be the same + + * "Is this reference monitor secure?" + + * "Does this reference monitor hamper performance?" + + * "Does this reference monitor prevent actions that should be allowed?" + +Notice that these questions are parallels of the security paradigms: security, efficiency and accuracy, respectively. + +If we can find a case where the hypothesis is false, then the security layer is not secure. Such a case is referred to as a counter example. Hence all test cases should be designed to test for these three types of flaws. + + +#### Information on: Try, Except, Else, Finally +The try, except, else and finally statements are part of **exception handling**. For more information on exception handling please visit: + + * [http://docs.python.org/tutorial/errors.html] + * [http://wiki.python.org/moin/HandlingExceptions] + * [http://www.tutorialspoint.com/python/python_exceptions.htm] + +### Hints and Ideas for testing + +When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: + + * threading + * multiple writes + +And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. + + +## Notes and Resources +---- + + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf + + * In repy 'log' replaces 'print' from python. Many students find this to be a stumbling block. + + * Note that you should not assume that any files exist in your directory. You should create any files (e.g., testfile.txt) yourself in your test program. + + * It's important to note that if a test has a flaw in any part of it, the entire test will be considered invalid. So, it's advisable to break your tests into different files. + +## How to run your tests on many reference monitors +---- + +Create a directory that the security layers will write their files into. You need to run repy with only access to this directory. You can write a test program that does `log(str(listfiles()))` to see if you are in the right place. + +Have all the reference monitors to test and the test cases inside the same directory where the repy.py file exists. + +* In the bash shell on Mac and Linux: +``` +for referencemonitor in reference_monitor_*; do for testcase in _*; do python repy.py restrictions.default encasementlib.r2py $referencemonitor $testcase; done; done +``` +* In the Command Prompt on Windows: +``` +FOR %r IN (reference_monitor_*) DO @FOR %a IN (_*) DO @python repy.py restrictions.default encasementlib.r2py %r %a +``` +* In PowerShell on Windows: +``` +foreach ($referencemonitor in Get-ChildItem reference_monitor_*) { foreach ($testcase in Get-ChildItem _*) { python repy.py restrictions.default encasementlib.r2py $referencemonitor.Name $testcase.Name } } +``` + +This will print out the output from each program. Make sure that you replace `` with your NetID. + +If you want to spot the referencemonitor that failed during the test run, add echo the name of each referencemonitor before the inner loop, like so: + +* In the bash shell on Mac and Linux: +``` +for referencemonitor in reference_monitor_*; do echo $referencemonitor under test; for testcase in _*; do python repy.py restrictions.default encasementlib.r2py $referencemonitor $testcase; done; done +``` +* In the Command Prompt on Windows: +``` +FOR %r IN (reference_monitor_*) DO @(ECHO %r under test & FOR %a IN (_*) DO @python repy.py restrictions.default encasementlib.r2py %r %a) +``` +* In PowerShell on Windows: +``` +foreach ($referencemonitor in Get-ChildItem reference_monitor_*) { Write-Host $referencemonitor.Name; foreach ($testcase in Get-ChildItem _*) { python repy.py restrictions.default encasementlib.r2py $referencemonitor.Name $testcase.Name } } +``` + +This will print out the name of each reference monitor before it starts executing the testcases against it. + + +## What to turn in? +---- + * Never raise unexpected errors or produce any output. Your attack must produce no output when run normally. + * Turn in the test cases used to attack a given reference monitor in a zip file. The name of each testcase must match the following format: [ net_id ]_attackcase1.r2py, [ net_id ]_attackcase2.r2py, etc. diff --git a/EducationalAssignments/ParityPartOne.md b/EducationalAssignments/ParityPartOne.md new file mode 100644 index 000000000..5c8b5c686 --- /dev/null +++ b/EducationalAssignments/ParityPartOne.md @@ -0,0 +1,328 @@ +# Implement a Defensive Security System + +This assignment will help you understand security mechanisms. You will be +guided through the steps of creating a reference monitor using the security +layer functionality in RepyV2. A reference monitor is an access control +concept that refers to an abstract machine that mediates all access to +objects by subjects. This can be used to allow, deny, or change the +behavior of any set of calls. While not a perfect way of validating your +reference monitor, it is useful to create test cases to see whether your +security layer will work as expected. (The test cases may be turned in as +part of the next assignment.) + +This assignment is intended to reinforce concepts about access control and +reference monitors in a hands-on manner. + + + + + + +## Overview +---- +In this assignment you will create a security layer which prevents writes that +do not maintain even parity for all 8 byte-aligned sequences in a file. +(What this means is described more precisely below.) Parity is often used to +detect errors or tampering for data. + +Your security layer will purely focus on storage of information in a file. +Using the minimal number of read blocks possible, you must determine if an +operation would change the parity. Your security layer must track enough +information about the parity of any 8-byte sequences read so that it need not +read the sequence more often than is necessary. + +All write operations must either complete or be blocked. All writes that would +not cause the parity of a 8-byte sequence to be non-even must be permitted. +Any write that would cause the parity of any 8-byte sequence to be non-even +must be blocked by throwing a RepyParityError exception. + +Note that in some cases there will be an incomplete sequence (e.g., the last +5 bytes of a 13 byte file). Parity is not checked for an incomplete sequence +(it is only checked when the sequence is completed). + +Note that the behavior of other system calls must not be changed in a way +that is visible to the running program. Reading from a file, opening a file, +etc. must appear to operate in the same manner. + +Three design paradigms are at work in this assignment: accuracy, +efficiency, and security. + + * Accuracy: The security layer should stop writeat calls if-and-only-if +they would result in non-even parity. All other actions should be allowed. +For example, if an app tries to read data a file or write data that results +in even parity 8-byte sequences these operations must succeed as per normal +and must not be blocked. All situations that are not described above *must* +match that of the underlying API. + + * Efficiency: The security layer should use a minimum number of resources, +so performance is not compromised. In particular, the security layer may not +read more 8-byte sequences than are necessary. Hint: it is *always* possible +to read two or fewer 8 byte blocks per writeat(). + + * Security: The attacker should not be able to circumvent the security +layer. Hence, if the attacker can cause a file with a non-even 8-byte sequence +to be written then the security is compromised, for example. + + +### Parity and 8-byte sequences + +For this assignment a file is conceptually broken up into 8-byte sequences. +Every consecutive series of 8 bytes (from the beginning of a file) is its own +8-byte sequence. So, the first 8 bytes (bytes 1-8) are the first sequence, the +next 8 bytes (bytes 9-16) are the second, etc. + +Note that a write may be performed on a non-8-byte-aligned portion of the +file. E.g., bytes 5-17 may be written in a single write. For that write, the +first, second, and third 8 byte sequence are all modified. + +In terms of parity, each byte has a parity based upon its value when calling +`ord()` in Python. Each byte that has a parity divisible by 2 is considered to +be even. An 8-byte sequence is considered to be even if there are an even +number of non-even bytes in the sequence. In other words, if there are 0, +2, 4, 6, or 8 non-even bytes, the sequence is considered to be even. Also, +if a sequence has not been completely written (because it is at the end of a +file), it is always considered to have even parity for the purposes of a +write being allowed or blocked. + + +### Getting Python and RepyV2 + +Please refer to the [SeattleTestbed Build Instructions](../Contributing/BuildInstructions.md#prerequisites) +for details. + +Once you have built RepyV2 into a directory of your choice, change into that +directory. Use the command below in order to run your RepyV2 programs: + +```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [program].r2py``` + +(Replace `[security_layer].r2py` and `[program].r2py` by the names of the +security layers and program that you want to run.) + +In order to test whether or not these steps worked, please copy and paste +the code found below for the sample security layer and sample attack. + +If you got an error, please go through the troubleshooting section found below. + +### Troubleshooting Repy code +---- +If you can't get Repy files to run, some of the following common errors may +have occurred: + + * using `print` instead of `log`: + +Repy is a subset of Python, but its syntax is slightly different. For +example, Python's `print` statement cannot be used; Repy has `log` for +that. For a full list of acceptable syntax please see +[https://github.com/SeattleTestbed/docs/blob/master/Programming/RepyV2API.md] + + * command line errors: + +**files are missing:** In the above command line call, you must have +`repy.py`, restrictions.default, encasementlib.r2py, the security layer and +the program you want to run in the current working directory. If any or +all of the above files are not in that directory then you will not be able +to run repy files. + + + + +### Tutorials for Repy and Python +---- +Now that you have Repy and Python, you may need a refresher on how to use +them. The following tutorials provide this information. + + * Official [Python tutorial](http://docs.python.org/tutorial/) + * [Differences between RepyV2 and Python](../Programming/PythonVsRepyV2.md) + * List of [RepyV2 API calls](../Programming/RepyV2API.md) + + + +## Building the security layer +---- +The following program is a basic and incomplete sample code for you to get +an idea about writing security layer. Remember, you have no idea how the +attacker will try to penetrate your security layer, so it is important that +you leave nothing to chance! + + +### A basic (and inadequate) defense + +Time to start coding! Let's inspect a basic security layer. + +``` +""" +This security layer inadequately handles parity for files in RepyV2. + + + +Note: + This security layer uses encasementlib.r2py, restrictions.default, repy.py and Python + Also you need to give it an application to run. + python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py + + """ + +class RepyParityError(Exception): + pass + +class EvenParityFile(): + def __init__(self, filename, create): + # globals + mycontext['debug'] = False + # local (per object) reference to the underlying file + self.fn = filename + + self.file = openfile(self.fn, create) + + + def writeat(self,data,offset): + + # check the parity of the data written + # NOTE: This is wrong in many ways!!!! + thisdata = data + while thisdata: + eightbytesequence = thisdata[:8] + thisdata = thisdata[8:] + even = True + for thisbyte in eightbytesequence: + # for each byte, if it is odd, flip even to be the opposite + if ord(thisbyte) % 2: + even = not even + + # actually call write, if we are supposed to... + if even: + self.file.writeat(eightbytesequence, offset) + # ...or error out. + else: + raise RepyParityError("Non-even parity write to file") + + + def readat(self, bytes, offset): + # Read from the file using the sandbox's readat... + return self.file.readat(bytes, offset) + + def close(self): + self.file.close() + + +def parityopenfile(filename, create): + return EvenParityFile(filename, create) + + + + +# The code here sets up type checking and variable hiding for you. You +# should not need to change anything below here. +sec_file_def = {"obj-type":EvenParityFile, + "name":"EvenParityFile", + "writeat":{"type":"func","args":(str,(int,long)),"exceptions":Exception,"return":(int,type(None)),"target":EvenParityFile.writeat}, + "readat":{"type":"func","args":((int,long,type(None)),(int,long)),"exceptions":Exception,"return":str,"target":EvenParityFile.readat}, + "close":{"type":"func","args":None,"exceptions":None,"return":(bool,type(None)),"target":EvenParityFile.close} + } + +CHILD_CONTEXT_DEF["openfile"] = {"type":"objc","args":(str,bool),"exceptions":Exception,"return":sec_file_def,"target":parityopenfile} +CHILD_CONTEXT_DEF["RepyParityError"] = {"type":"any","target":RepyParityError} +# Execute the user code +secure_dispatch_module() +``` + + + +### Testing your security layer +---- +In this part of the assignment you will pretend to be an attacker. Remember +the attacker's objective is to bypass the parity restrictions or cause +the security layer to act in a disallowed manner. By understanding how the +attacker thinks, you will be able to write better security layers. + +An example of a test / attack is found below: + +``` +if "testfile.txt" in listfiles(): + removefile("testfile.txt") + +myfile=openfile("testfile.txt",True) #Create a parity file + +# Put some valid data in the file. +myfile.writeat("AA", 0) + +# I should be able to read it out. +assert('AA' == myfile.readat(None, 0)) + +# However, this write should fail... +try: + myfile.writeat("BCBCBC", 2) +except RepyParityError: + pass # should happen +else: + log("should have been an error instead!") + +# Close the file +myfile.close() + + +``` + +**Note:** All attacks should be written as RepyV2 files, using the `.r2py` extension. + +#### Choice of File Names +---- +It is important to keep in mind that only lowercase file names are allowed. +So in the above code, specifically: + +``` +# Open a file +myfile=openfile("look.txt", True) +``` + +`look.txt` is a valid file name, however `Look.txt` and `LOOK.TXT` are not. +Examples of other invalid files names are `.look.txt`, `look/.txt`, and +`look().txt`. Essentially all non-alphanumeric characters are not allowed. + +### Running your security layer +---- +Finally, type the following commands at the terminal to run your security +layer with your attack program + +```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py ``` + +Make sure you went through the "How to get RepyV2" section! + + +# Notes and Resources +---- + + * For a complete list of syntax in Repyv2 please visit + the **[RepyV2 API documentation](https://github.com/SeattleTestbed/docs/blob/master/Programming/RepyV2API.md)** + + * **[This paper](https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf)** + is an excellent source for information about security layers + + * **Note:** It is possible to add multiple security layers to Repy, this +may be useful for testing different mitigations separately. This is +done with the following command at the terminal: + +```python repy.py restrictions.default encasementlib.r2py [security_layer1].r2py [security_layer2].r2py [security_layer3].r2py [program].r2py``` + +**Your security layer must produce no output!!** + + * In RepyV2, `log` replaces `print` from Python. This may be helpful when +testing if Repy installed correctly. + +# What to turn in? +---- + * Turn in a repy file called `reference_monitor_netid.r2py` with all +letters in lowercase. For example, if your netId is `jc123`, your reference monitor should be named `reference_monitor_jc123.r2py` + +* **Never raise unexpected errors or produce any output.** Your program +must produce no output when run normally. diff --git a/EducationalAssignments/ParityPartThree.md b/EducationalAssignments/ParityPartThree.md index d0cf34098..a8803024a 100644 --- a/EducationalAssignments/ParityPartThree.md +++ b/EducationalAssignments/ParityPartThree.md @@ -1,7 +1,7 @@ # Fixing your security layer -In this assignment you will analyze the bugs in your security layer from [[Part One](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/ParityPartOne.md)] and fix them. You may want to use test cases from [Part Two](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/ParityPartTwo.md) to help identify these bugs. Finally, you will write a report discussing the different classes of bugs that your code had, and why. +In this assignment you will analyze the bugs in your security layer from [Part One](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/ParityPartOne.md) and fix them. You may want to use test cases from [Part Two](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/ParityPartTwo.md) to help identify these bugs. Finally, you will write a report discussing the different classes of bugs that your code had, and why. @@ -52,8 +52,8 @@ Let's analyze one of the bugs in this implementation. According to the specific ## What to turn in? ---- - * Turn in the reference monitor that you have fixed. The name of the reference monitor should be in the form of reference_monitor_[poly_username].r2py + * Turn in the reference monitor that you have fixed. The name of the reference monitor should be in the form of reference_monitor_netId.r2py e.g. reference_monitor_jcappos.r2py All letters must be lowercase. - * In addition to your reference monitor, submit a one-page PDF document, in which you discuss the different classes of bugs you had in your reference monitors and why. + * In addition to your reference monitor, submit the one-time form whose link is shared in the assignments tab, in which you discuss the different classes of bugs you had in your reference monitors and why. What kind of bug did you experience, what caused the bug, and what did you do to fix it? diff --git a/EducationalAssignments/ParityPartTwo.md b/EducationalAssignments/ParityPartTwo.md index 14cd4f668..67349e0cf 100644 --- a/EducationalAssignments/ParityPartTwo.md +++ b/EducationalAssignments/ParityPartTwo.md @@ -149,7 +149,7 @@ The try, except, else and finally statements are part of **exception handling**. When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: * threading - * writing to multiple files???? + * writing to multiple files * multiple writes And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. @@ -158,7 +158,7 @@ And more! Remember a good security layer can't be broken by anyone! Which is a ## Notes and Resources ---- - * The following link is an excellent source for information about security layers: http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf * [repy_v2/benchmarking-support/allnoopsec.py](https://seattle.poly.edu/browser/seattle/branches/repy_v2/benchmarking-support/allnoopsec.py) is an empty security layer that doesn't perform any operations. diff --git a/EducationalAssignments/PermissionsPartOne.md b/EducationalAssignments/PermissionsPartOne.md index 93fa5e20d..31fcc9fce 100644 --- a/EducationalAssignments/PermissionsPartOne.md +++ b/EducationalAssignments/PermissionsPartOne.md @@ -1,4 +1,4 @@ -# Building a Security Layer +# Building a Security Layer This assignment will help you understand security mechanisms. You will be guided through the steps of creating a reference monitor using the security layer functionality in Repy V2. A reference monitor is an access control concept that refers to an abstract machine that mediates all access to objects by subjects. This can be used to allow, deny, or change the behavior of any set of calls. One critical aspect of creating a reference monitor is to ensure it cannot be bypassed. While not a perfect solution, it is useful to create test cases to see whether your security layer will work as expected. (The test cases may be turned in as part of the next assignment) @@ -7,9 +7,9 @@ This assignment is intended to prepare you for thinking about security paradigms - -## Overview + +## Overview ---- In this assignment you will create a security layer which stops the user from reading data, writing over existing data, or writing new data to the end of a file without having the permission to do so. You will write five functions: setread, setwrite, setappend, readat, and writeat. The setread, setwrite, and setappend functions enable and disable read, write, and append permissions, respectively. The user may have all, some, or none of the permissions enabled at any given time. By default, these permissions are disabled. The readat function will allow the user to read data from the file if and only if the user has read permissions. The writeat function will allow the user to overwrite existing data if and only if write permissions are enabled. Similarly, the user may append new data to the file if and only if append permissions are enabled. Attempting to overwrite or append data when the user does not have permissions to do so will result in nothing being written. If both append and write permissions are disabled when a writeat is called, a ValueError is raised. You will do this by adding security rules to the functions available for setting the permissions, as well as reading from and writing to a file.Think of this as a method to write some data securely into a file. The future of the system depends on your ability to write secure code! @@ -21,15 +21,15 @@ Three design paradigms are at work in this assignment: accuracy, efficiency, and * Security: The attacker should not be able to circumvent the security layer. Hence, if the data can be written without permission, the security is compromised. - -### Getting Python + +### Getting Python ---- Please note you must have Python 2.5, 2.6, or 2.7 to complete this assignment. Instructions on how to get Python for Windows can be found [InstallPythonOnWindows here]. If you are using Linux or a Mac it is likely you already have Python. In order to verify this, simply open a terminal and type ```python```. Please check its version on the initial prompt. **Note:**If you are using Windows, you will need python in your path variables. A tutorial on adding variables to your path in Windows can be found at [http://www.computerhope.com/issues/ch000549.htm] -### Getting RepyV2 +### Getting RepyV2 ---- The preferred way to get Repy is ''installing from source''. For this, you check out the required Git repositories, and run a build script. You can always update the repos later, and rebuild, so that you get the latest stable version of the Repy runtime. @@ -70,7 +70,7 @@ In order to test whether or not these steps worked, please copy and paste the co If you got an error, please go through the trouble shooting section found below. -### Troubleshooting Repy code +### Troubleshooting Repy code If you can't get Repy files to run, some of the following common errors may have occurred: * using `print` instead of `log`: @@ -94,24 +94,24 @@ To run the unit test, which will automatically tell you if you have errors with * [wiki:RepyV2CheckoutAndUnitTests] --> - -### Tutorials for Repy and Python + +### Tutorials for Repy and Python ---- Now that you have Repy and Python, you may need a refresher on how to use them. The following tutorials are excellent sources of information. * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Building the security layer + +## Building the security layer ---- **[wiki:RepyV2SecurityLayers]** explains the syntax used to build a reference monitor. The general tutorials above will aid in looking up other details about Repy. Remember, you have no idea how the attacker will try to penetrate your security layer, so it is important that you leave nothing to chance! Your reference monitor should try to stop every attack you can think of. Not just one or two. A good reference monitor will do this and incorporate the design paradigms of accuracy, efficiency and security defined above. - -### A basic (and inadequate) defense + +### A basic (and inadequate) defense Time to start coding! Let's inspect a basic security layer. @@ -218,19 +218,19 @@ secure_dispatch_module() ``` -### Using the example layer +### Using the example layer Keep in mind the above security layer would not prevent writing without permissions, and only protects reading without permissions from certain attacks. There are a bunch of tricks that can be used in order to circumvent this security layer easily. For instance, the above reference monitor isn't thread safe. For an introduction to thread safety please read [wiki/Thread_safety](http://en.wikipedia.org/wiki/Thread_safety). - -### Code analysis + +### Code analysis The above functions attempt to follow the design principles of accuracy, efficiency, and security. However they do so inadequately.For example the data will not be written at all unless both permissions to write and append are enabled. It is more important to make a security system that is infeasible to break, rather than impossible to break. However this security layer does not yet create the necessary infeasibility requirements for most attacks. Thus we see that there is a grey area of what is an acceptable level of impedance. - -### Testing your security layer + +### Testing your security layer ---- In this part of the assignment you will pretend to be an attacker. Remember the attacker's objective is to bypass permissions. By understanding how the attacker thinks, you will be able to write better security layers. Perhaps while attacking your security layer you will think of a new mitigation that should have been implemented. Keep in mind attacks are attempts to mitigate a given security protocol. If even one case succeeds, then your security layer has been compromised. Thus the attack you write should include several methods of attempting to bypass permissions. An example of an attack is found below: @@ -266,7 +266,7 @@ myfile.close() **Note:** All attacks should be written as Repy files, using the .r2py extension. -#### Code Analysis +#### Code Analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -279,16 +279,16 @@ look.txt is a valid file name, however Look.txt is not. Examples of other invali This code attempts to read the “This is secure” from the file directly. First the file is opened using myfile=openfile("look.txt",True). Next myfile.writeat writes some data to the file with permissions. Then the write permissions are removed, and writeat is once again used to write some data to the file, some of which attempts to overwrite the existing data. If the security layer is written properly, only the appended data will be written. -### Running your security layer +### Running your security layer ---- Finally, type the following commands at the terminal to run your security layer with your attack program ```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py ``` Make sure you went through the "How to get RepyV2" section! - -# Notes and Resources + +# Notes and Resources ---- * A list of command line utilities for windows can be found at **[http://commandwindows.com/command3.htm]** @@ -299,7 +299,7 @@ Make sure you went through the "How to get RepyV2" section! * For a complete list of syntax in Repyv2 please visit: **[wiki:RepyV2API]** - * The following link is an excellent source for information about security layers: **[http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf]** + * The following link is an excellent source for information about security layers: **[https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf]** * **[repy_v2/benchmarking-support/allnoopsec.py](https://seattle.poly.edu/browser/seattle/branches/repy_v2/benchmarking-support/allnoopsec.py)** is an empty security layer that doesn't perform any operations. @@ -312,15 +312,15 @@ Make sure you went through the "How to get RepyV2" section! **Your security layer should produce no output!! **If it encounters an attempt to read the secure data, you should raise a ValueError exception and allow execution to continue. * In repy log replaces print from python. This may be helpful when testing if Repy installed correctly. - -# Extra Credit + +# Extra Credit ---- Remember permissions for files after they are closed and reopened. This would include situations where your security layer is itself closed and reopened (for example, if the system restarts). You can assume that all reads must go through your security layer and so you can add an additional file or modify the file format. - -# What to turn in? + +# What to turn in? ---- * Turn in a repy file called reference_monitor_[ polyusername ].r2py with all letters in lowercase. Be sure your reference monitor never produces output. It should raise a ValueError if the calls should be blocked, but never, ever call log() to output information. Never raise unexpected errors. diff --git a/EducationalAssignments/PermissionsPartTwo.md b/EducationalAssignments/PermissionsPartTwo.md index 38cad96dd..616b1ade1 100644 --- a/EducationalAssignments/PermissionsPartTwo.md +++ b/EducationalAssignments/PermissionsPartTwo.md @@ -1,12 +1,12 @@ -# Security layer testing and penetration +# Security layer testing and penetration In this assignment you will learn how to attack a reference monitor. The reference monitor you will be testing uses the security layer framework (encasement library, etc.) for the Seattle testbed. It is possible to do this assignment separately however it is recommended that this assignment be completed after [wiki:EducationalAssignments/PermissionsPartOne PermissionsPartOne]. Either way you should already have a working security layer or access to one. Testing the security layer is done by running a series of test cases that an adversary may use to circumvent your system. This assignment is intended to prepare you for thinking about security paradigms in a functional way. The ideas of information, security and privacy have been embedded into the steps of this assignment. - -## Overview + +## Overview ---- In this assignment you are a tester. You have been sent a bunch of reference monitors that need testing before they are deployed. Your job will be to ensure an attacker cannot circumvent these security layers. In order to do this, you will attempt to read, write, and append data without having the proper missions enabled. If you are able to do so, then the security layer is not secure. The future of the system depends on your ability to test code thoroughly! @@ -28,24 +28,24 @@ Within the context of this assignment these design paradigms translate to: You will submit a zip file containing all of the tests you have created. You will gain points for every student's reference monitor you find a flaw in. It is okay if multiple tests of yours breaking a student's reference monitor, but you will not gain additional points. - -## Prerequisites + +## Prerequisites This assignment assumes you have Python2.5 or Python2.6, Repy and RepyV2 installed on your computer. If you don't already have them please go to [wiki:EducationalAssignments/PermissionsPartOne#GettingPythonRepy PermissionsPartOne] for a tutorial on how to get them. - -### Helpful links + +### Helpful links ---- The following links will aid students in becoming comfortable with Python, Repy and seattle: * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Testing security layers + +## Testing security layers ---- -### Hypothesis, test case, counter example +### Hypothesis, test case, counter example The goal of a good tester is to test hypotheses. A hypothesis is just a scientific way of asking a question. The hypothesis of this assignment is "This security layer is well designed." The questions you will ask when running your test cases will always be the same @@ -59,11 +59,11 @@ Notice that these questions are parallels of the security paradigms: security, e If we can find a case where the hypothesis is false, then the security layer is not secure. Such a case is referred to as a counter example. Hence all test cases should be designed to test for these three types of flaws. - -### Examples of tests + +### Examples of tests Test cases are briefly described in [wiki:EducationalAssignments/PermissionsPartOne PermissionsPartOne] and [wiki:RepyV2SecurityLayers]. Below is another example of a test case you may want to consider. This test case gives the right 'style' for your all your test cases, but lacks in the number of test cases. A good attack will include many test cases. -#### Test case 1: +#### Test case 1: ``` # Open a file @@ -91,7 +91,7 @@ finally: # Close the file after our attempt. myfile.close() ``` -#### Code analysis +#### Code analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -104,7 +104,7 @@ look.txt is a valid file name, however Look.txt is not. Examples of other inval In this case we are verifying the security of the reference monitor. This code attempts to append data without the proper permissions to do so. First the file is opened using `myfile=openfile("look.txt",True)`. Next `myfile.readat(1,0)` tries to read from the file. The 0 refers to an offset of zero and 1 refers to number of characters being read. The try: statement tells the program to "try" this case. Notice that the except is executed if an error is raised. If the security layer fails the test then the else statement is executed. The finally: statement will always run, closing the file. -#### Test case 2: +#### Test case 2: ``` # Open a file @@ -128,20 +128,20 @@ finally: # Close the file after our attempt. myfile.close() ``` -#### Code analysis +#### Code analysis In this case we are verifying the accuracy of the reference monitor. This code attempts to write `"abcd"` to the file directly. We assume that secure data is written in the file on the first four offset values. First the file is opened using `myfile=openfile("look.txt",True)`, and the append and read permissions are enabled. Next `myfile.writeat("abcd",0)` tries to write `"abcd"` to the file. The 0 refers to an offset of zero. The try: statement tells the program to "try" this case. Notice that the except is executed if an error is raised. If the security layer fails the test then the else statement is executed. The finally: statement will always run, closing the file. If this case produces anything other than "Write okay", then this layer fails the accuracy design paradigm. The security layer should only stop a file from reading the secure data not overwriting it. -#### More information on: Try, Except, Else, Finally +#### More information on: Try, Except, Else, Finally The try, except, else and finally statements are part of **exception handling**. For more information on exception handling please visit: * [http://docs.python.org/tutorial/errors.html] * [http://wiki.python.org/moin/HandlingExceptions] * [http://www.tutorialspoint.com/python/python_exceptions.htm] -### Hints and Ideas for testing +### Hints and Ideas for testing When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: @@ -149,12 +149,12 @@ When writing your own tests it is important to test for a complete set of possib * writing to multiple files And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. - -## Notes and Resources + +## Notes and Resources ---- - * The following link is an excellent source for information about security layers: http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf * [repy_v2/benchmarking-support/allnoopsec.py](https://seattle.poly.edu/browser/seattle/branches/repy_v2/benchmarking-support/allnoopsec.py) is an empty security layer that doesn't perform any operations. @@ -163,15 +163,15 @@ And more! Remember a good security layer can't be broken by anyone! Which is a * In repy 'log' replaces 'print' from python. Many students find this to be a stumbling block. - -## Extra Credit + +## Extra Credit ---- Find bugs in the extra credit reference monitors given the altered threat model. You should include more test cases in the extra credit! - -## How to run your tests on many reference monitors + +## How to run your tests on many reference monitors ---- If you are using Mac or Linux, you can do something like the following: @@ -188,10 +188,10 @@ This will print out the output from each program. If you are a Windows user and you create your own solution that does the same thing, please post it on the forum. - -## What to turn in? + +## What to turn in? ---- * Turn in the test cases used to attack a given reference monitor in a zip file. The name of each testcase must start with your poly email id in lowercase. For example: jcappos_securitytest1.repy justincappos_goodaccuracytest.repy are both valid names. - * Optionally turn in the test cases used to attack the extra credit reference monitors in a zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: jcappos_run_twice_metadata_removal.repy jcappos_run_once_threading_hack.repy. \ No newline at end of file + * Optionally turn in the test cases used to attack the extra credit reference monitors in a zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: jcappos_run_twice_metadata_removal.repy jcappos_run_once_threading_hack.repy. diff --git a/EducationalAssignments/PrivateWritePartOne.md b/EducationalAssignments/PrivateWritePartOne.md index 9ff931f03..e2ed95ed9 100644 --- a/EducationalAssignments/PrivateWritePartOne.md +++ b/EducationalAssignments/PrivateWritePartOne.md @@ -1,5 +1,5 @@ -# Building a Security Layer +# Building a Security Layer This assignment will help you understand security mechanisms. You will be guided through the steps of creating a reference monitor using the security layer functionality in Repy V2. A reference monitor is an access control concept that refers to an abstract machine @@ -13,9 +13,9 @@ This assignment is intended to prepare you for thinking about security paradigms - -## Overview + +## Overview ---- In this assignment you will create a security layer which stops the attacker from reading data securely written in a file.You will create a function called **privatewrite** which allows the user to write data into a file which cannot be read back from the file.The data can be overwritten but isn't readable under any circumstance. You will do this by adding security rules to the functions available for reading from and writing to a file.Think of this as a method to write some data securely into a file. The future of the system depends on your ability to write secure code! @@ -26,15 +26,15 @@ Three design paradigms are at work in this assignment: accuracy, efficiency, and * Efficiency: The security layer should use a minimum number of resources, so performance is not compromised. This means you should not do things like re-read the file before each write. * Security: The attacker should not be able to circumvent the security layer. Hence, if the data written using privatewrite can be read back, the security is compromised. - -### Getting Python + +### Getting Python ---- Please note you must have Python 2.5, 2.6, or 2.7 to complete this assignment. Instructions on how to get Python for Windows can be found [InstallPythonOnWindows here]. If you are using Linux or a Mac it is likely you already have Python. In order to verify this, simply open a terminal and type ```python```. Please check its version on the initial prompt. **Note:**If you are using Windows, you will need python in your path variables. A tutorial on adding variables to your path in Windows can be found at [http://www.computerhope.com/issues/ch000549.htm] -### Getting RepyV2 +### Getting RepyV2 ---- The preferred way to get Repy is ''installing from source''. For this, you check out the required Git repositories, and run a build script. You can always update the repos later, and rebuild, so that you get the latest stable version of the Repy runtime. @@ -75,7 +75,7 @@ In order to test whether or not these steps worked, please copy and paste the co If you got an error, please go through the trouble shooting section found below. -### Trouble shooting Repy code +### Trouble shooting Repy code If you can't get Repy files to run, some of the following common errors may have occurred: * using `print` instead of `log`: @@ -99,24 +99,24 @@ To run the unit test, which will automatically tell you if you have errors with * [wiki:RepyV2CheckoutAndUnitTests] --> - -### Tutorials for Repy and Python + +### Tutorials for Repy and Python ---- Now that you have Repy and Python, you may need a refresher on how to use them. The following tutorials are excellent sources of information. * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Building the security layer + +## Building the security layer ---- **[wiki:RepyV2SecurityLayers]** explains the syntax used to build a reference monitor. The general tutorials above will aid in looking up other details about Repy. Remember, you have no idea how the attacker will try to penetrate your security layer, so it is important that you leave nothing to chance! Your reference monitor should try to stop every attack you can think of. Not just one or two. A good reference monitor will do this and incorporate the design paradigms of accuracy, efficiency and security defined above. - -### A basic (and inadequate) defense + +### A basic (and inadequate) defense Time to start coding! Let's inspect a basic security layer. @@ -183,22 +183,22 @@ CHILD_CONTEXT_DEF["openfile"] = {TYPE:OBJC,ARGS:(str,bool),EXCP:Exception,RETURN secure_dispatch_module() ``` -### Using the example layer +### Using the example layer Keep in mind the above security layer would only stop one kind of attack. Thus if an attacker doesn't know much about file input/output this will probably stop them. However there are a bunch of tricks that can be used in order to circumvent this security layer easily. For instance, the above reference monitor isn't thread safe. For an introduction to thread safety please read [wiki/Thread_safety](http://en.wikipedia.org/wiki/Thread_safety). - -### Code analysis + +### Code analysis The `privatewrite()` function attempts to follow the design principles of accuracy, efficiency, and security. However it does so inadequately.For example the data is blocked even if its overwritten using a normal `writeat()` function. It is more important to make a security system that is infeasible to break, rather than impossible to break. However this security layer does not yet create the necessary infeasibility requirements for most attacks. Thus we see that there is a grey area of what is an acceptable level of impedance. Looking at the structure of `privatewrite()`,the flag `privatedata` is set when data is written using `privatewrite()`. During the `readat()` function, the flag is checked against and accordingly the data is either displayed or an exception is raised. - -### Testing your security layer + +### Testing your security layer ---- In this part of the assignment you will pretend to be an attacker. Remember the attacker's objective is to read the `secure data` written in the file. By understanding how the attacker thinks, you will be able to write better security layers. Perhaps while attacking your security layer you will think of a new mitigation that should have been implemented. Keep in mind attacks are attempts to mitigate a given security protocol. If even one case succeeds, then your security layer has been compromised. Thus the attack you write should include several methods of attempting to read the secure data from the file. An example of an attack is found below: ``` @@ -233,7 +233,7 @@ finally: **Note:** All attacks should be written as Repy files, using the `.r2py` extension. -#### Code Analysis +#### Code Analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -245,18 +245,18 @@ myfile=openfile("look.txt",True) look.txt is a valid file name, however Look.txt is not. Examples of other invalid files names are, look@.txt, look/.txt, and look().txt. Essentially all non-alphanumeric characters are not allowed. This code attempts to read the `This is secure` from the file directly. First the file is opened using -`myfile=openfile("look.txt",True)`. Next `myfile.writeat` writes some data to the file. Then the `privatewrite` function is used to write some data securely to the file. The 0 refers to an offset of zero. The `try:` statement tells the program to "try" this case. Notice that the `except` is executed if an error is raised. If the security layer fails the test then the else statement is executed. The `finally:` statement will always run, closing the file. +`myfile=openfile("look.txt",True)`. Next `myfile.writeat` writes some data to the file. Then the `privatewrite` function is used to write some data securely to the file. The 0 refers to an offset of zero. The `try:` statement tells the program to "try" this case. Notice that the `except` is executed if an error is raised. If the security layer fails the test then the else statement is executed. The `finally:` statement will always run, closing the file. -### Running your security layer +### Running your security layer ---- Finally, type the following commands at the terminal to run your security layer with your attack program ```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py ``` Make sure you went through the "How to get RepyV2" section! - -# Notes and Resources + +# Notes and Resources ---- * A list of command line utilities for windows can be found at **[http://commandwindows.com/command3.htm]** @@ -267,7 +267,7 @@ Make sure you went through the "How to get RepyV2" section! * For a complete list of syntax in Repyv2 please visit: **[wiki:RepyV2API]** - * The following link is an excellent source for information about security layers: **[http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf]** + * The following link is an excellent source for information about security layers: **[https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf]** * **[repy_v2/benchmarking-support/allnoopsec.py](https://seattle.poly.edu/browser/seattle/branches/repy_v2/benchmarking-support/allnoopsec.py)** is an empty security layer that doesn't perform any operations. @@ -280,17 +280,17 @@ Make sure you went through the "How to get RepyV2" section! **Your security layer should produce no output!! **If it encounters an attempt to read the secure data, you should raise a `ValueError` exception and allow execution to continue. * In repy log replaces print from python. This may be helpful when testing if Repy installed correctly. - -# Extra Credit + +# Extra Credit ---- For extra credit - Protect the file against read and write operations involving persistent data. If you close the file and open it again the system should be able to block the future attacks. - -# What to turn in? + +# What to turn in? ---- * Turn in a repy file called reference_monitor_[ name or id number].r2py. Be sure your reference monitor **never produces output**. It should raise a ValueError if the readat should be blocked, but never, ever call log() to output information or raise unexpected errors. - * For extra credit turn in a second repy file called extra_credit_[ name or id number].r2py \ No newline at end of file + * For extra credit turn in a second repy file called extra_credit_[ name or id number].r2py diff --git a/EducationalAssignments/PrivateWritePartTwo.md b/EducationalAssignments/PrivateWritePartTwo.md index 3603c9f63..3ef7eb4da 100644 --- a/EducationalAssignments/PrivateWritePartTwo.md +++ b/EducationalAssignments/PrivateWritePartTwo.md @@ -1,12 +1,12 @@ -# Security layer testing and penetration +# Security layer testing and penetration In this assignment you will learn how to attack a reference monitor. The reference monitor you will be testing uses the security layer framework (encasement library, etc.) for the Seattle testbed. It is possible to do this assignment separately however it is recommended that this assignment be completed after [wiki:EducationalAssignments/PrivateWritePartOne PrivateWritePartOne]. Either way you should already have a working security layer or access to one. Testing the security layer is done by running a series of test cases that an adversary may use to circumvent your system. This assignment is intended to prepare you for thinking about security paradigms in a functional way. The ideas of information, security and privacy have been embedded into the steps of this assignment. - -## Overview + +## Overview ---- In this assignment you are a tester. You have been sent a bunch of reference monitors that need testing before they are deployed. Your job will be to ensure an attacker cannot circumvent these security layers. In order to do this, you will attempt to read the secure data written in the file using the function `privatewrite()`. If you are able to do so, then the security layer is not secure. The future of the system depends on your ability to test code thoroughly! @@ -28,24 +28,24 @@ Within the context of this assignment these design paradigms translate to: You will submit a zip file containing all of the tests you have created. You will gain points for every student's reference monitor you find a flaw in. It is okay if multiple tests of yours breaking a student's reference monitor, but you will not gain additional points. - -## Prerequisites + +## Prerequisites This assignment assumes you have Python2.5 or Python2.6, Repy and RepyV2 installed on your computer. If you don't already have them please go to [wiki:EducationalAssignments/PrivateWritePartOne#GettingPythonRepy PrivateWritePartOne] for a tutorial on how to get them. - -### Helpful links + +### Helpful links ---- The following links will aid students in becoming comfortable with Python, Repy and seattle: * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Testing security layers + +## Testing security layers ---- -### Hypothesis, test case, counter example +### Hypothesis, test case, counter example The goal of a good tester is to test hypotheses. A hypothesis is just a scientific way of asking a question. The hypothesis of this assignment is "This security layer is well designed." The questions you will ask when running your test cases will always be the same @@ -59,11 +59,11 @@ Notice that these questions are parallels of the security paradigms: security, e If we can find a case where the hypothesis is false, then the security layer is not secure. Such a case is referred to as a counter example. Hence all test cases should be designed to test for these three types of flaws. - -### Examples of tests + +### Examples of tests Test cases are briefly described in [wiki:EducationalAssignments/PrivateWritePartOne PrivateWritePartOne] and [wiki:RepyV2SecurityLayers]. Below is another example of a test case you may want to consider. This test case gives the right 'style' for your all your test cases, but lacks in the number of test cases. A good attack will include many test cases. -#### Test case 1: +#### Test case 1: ``` # Open a file @@ -88,7 +88,7 @@ finally: # Close the file after our attempt. myfile.close() ``` -#### Code analysis +#### Code analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -101,7 +101,7 @@ look.txt is a valid file name, however Look.txt is not. Examples of other inval In this case we are verifying the security of the reference monitor. This code attempts to read the data written on the first offset value. First the file is opened using `myfile=openfile("look.txt",True)`. Next `myfile.readat(1,0)` tries to read from the file. The 0 refers to an offset of zero and 1 refers to number of characters being read. The try: statement tells the program to "try" this case. Notice that the except is executed if an error is raised. If the security layer fails the test then the else statement is executed. The finally: statement will always run, closing the file. -#### Test case 2: +#### Test case 2: ``` # Open a file @@ -124,20 +124,20 @@ finally: # Close the file after our attempt. myfile.close() ``` -#### Code analysis +#### Code analysis In this case we are verifying the accuracy of the reference monitor. This code attempts to write `"abcd"` to the file directly. We assume that secure data is written in the file on the first four offset values. First the file is opened using `myfile=openfile("look.txt",True)`. Next `myfile.writeat("abcd",0)` tries to write `"abcd"` to the file. The 0 refers to an offset of zero. The try: statement tells the program to "try" this case. Notice that the except is executed if an error is raised. If the security layer fails the test then the else statement is executed. The finally: statement will always run, closing the file. If this case produces anything other than "Write okay", then this layer fails the accuracy design paradigm. The security layer should only stop a file from reading the secure data not overwriting it. -#### More information on: Try, Except, Else, Finally +#### More information on: Try, Except, Else, Finally The try, except, else and finally statements are part of **exception handling**. For more information on exception handling please visit: * [http://docs.python.org/tutorial/errors.html] * [http://wiki.python.org/moin/HandlingExceptions] * [http://www.tutorialspoint.com/python/python_exceptions.htm] -### Hints and Ideas for testing +### Hints and Ideas for testing When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: @@ -146,12 +146,12 @@ When writing your own tests it is important to test for a complete set of possib * writing to both positions at once And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. - -## Notes and Resources + +## Notes and Resources ---- - * The following link is an excellent source for information about security layers: http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf * [repy_v2/benchmarking-support/allnoopsec.py](https://seattle.poly.edu/browser/seattle/branches/repy_v2/benchmarking-support/allnoopsec.py) is an empty security layer that doesn't perform any operations. @@ -160,15 +160,15 @@ And more! Remember a good security layer can't be broken by anyone! Which is a * In repy 'log' replaces 'print' from python. Many students find this to be a stumbling block. - -## Extra Credit + +## Extra Credit ---- Find bugs in the extra credit reference monitors given the altered threat model. You should include more test cases in the extra credit! - -## How to run your tests on many reference monitors + +## How to run your tests on many reference monitors ---- If you are using Mac or Linux, you can do something like the following: @@ -185,11 +185,11 @@ This will print out the output from each program. If you are a Windows user and you create your own solution that does the same thing, please post it on the forum. - -## What to turn in? + +## What to turn in? ---- * Turn in the test cases used to attack a given reference monitor in a zip file. The name of each testcase must start with your first and last name in lowercase separated by underscores: first_last_. For example: justin_cappos_securitytest1.repy justincappos_goodaccuracytest.repy are both valid names. * Optionally turn in the test cases used to attack the extra credit reference monitors in a zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: justin_cappos_run_twice_metadata_removal.repy justincappos_run_once_threading_hack.repy. - \ No newline at end of file + diff --git a/EducationalAssignments/ProtectFilePartOne.md b/EducationalAssignments/ProtectFilePartOne.md index 28dd6c917..eeff5de5f 100644 --- a/EducationalAssignments/ProtectFilePartOne.md +++ b/EducationalAssignments/ProtectFilePartOne.md @@ -1,4 +1,4 @@ -# Implement a Defensive Security System +# Implement a Defensive Security System This assignment will help you understand security mechanisms. You will be guided through the steps of creating a reference monitor using the security layer functionality in Repy V2. A reference monitor is an access control concept that refers to an abstract machine that mediates all access to objects by subjects. This can be used to allow, deny, or change the behavior of any set of calls. While not a perfect way of validating your reference monitor, it is useful to create test cases to see whether your security layer will work as expected. (The test cases may be turned in as part of the next assignment.) @@ -7,9 +7,9 @@ This assignment is intended to reinforce concepts about access control and refer - -## Overview + +## Overview ---- In this assignment you will create a reference monitor that will, when asked to do so, protect files starting with private_ (and only those files). If the contents of a file begin with the characters “INV”, then a listfiles() call **must not** show that the file exists. If the contents of a file begin with the characters “PER” then a removefile() call **must** raise a RepyArgumentError. In cases not specified above, all calls must work identically to the RepyV2 API. @@ -20,18 +20,18 @@ Three design paradigms are at work in this assignment: accuracy, efficiency, and * Security: The attacker should not be able to circumvent the reference monitor and cause files that should not be displayed to be displayed or files that should not be removed to be removed. - -### Getting Python + +### Getting Python ---- Please note you must have Python 2.5, 2.6, or 2.7 to complete this assignment. Instructions on how to get Python for Windows can be found [InstallPythonOnWindows here]. If you are using Linux or a Mac it is likely you already have Python. In order to verify this, simply open a terminal and type ```python```. Please check its version on the initial prompt. **Note:**If you are using Windows, you will need python in your path variables. A tutorial on adding variables to your path in Windows can be found at [http://www.computerhope.com/issues/ch000549.htm] - -### Getting RepyV2 + +### Getting RepyV2 ---- The preferred way to get Repy is ''installing from source''. For this, you check out the required Git repositories, and run a build script. You can always update the repos later, and rebuild, so that you get the latest stable version of the Repy runtime. @@ -73,9 +73,9 @@ In order to test whether or not these steps worked, please copy and paste the co If you got an error, please go through the troubleshooting section found below. - -### Troubleshooting Repy code + +### Troubleshooting Repy code ---- If you can't get Repy files to run, some of the following common errors may have occurred: @@ -102,9 +102,9 @@ To run the unit test, which will automatically tell you if you have errors with --> - -### Tutorials for Repy and Python + +### Tutorials for Repy and Python ---- Now that you have Repy and Python, you may need a refresher on how to use them. The following tutorials are excellent sources of information. @@ -113,16 +113,16 @@ Now that you have Repy and Python, you may need a refresher on how to use them. * List of RepyV2 API calls: **[wiki:RepyV2API]** - -## Building the security layer + +## Building the security layer ---- The following program is a basic and incomplete sample code for you to get an idea about writing security layer. Remember, you have no idea how the attacker will try to penetrate your security layer, so it is important that you leave nothing to chance! - -### A basic (and inadequate) defense + +### A basic (and inadequate) defense ---- Time to start coding! Let's inspect a basic security layer. @@ -168,7 +168,7 @@ def secure_openfile(filename, create): def secure_removefile(filename): if filename in PERBUFFER: - raise RepyArgumentError("Cannot delete this file! + raise RepyArgumentError("Cannot delete this file! n") removefile(filename) @@ -219,9 +219,9 @@ secure_dispatch_module() ``` - -### Testing your security layer + +### Testing your security layer ---- In this part of the assignment you will pretend to be an attacker. Remember the attacker's objective is to bypass the restrictions or cause the security layer to act in a disallowed manner. By understanding how the attacker thinks, you will be able to write better security layers. @@ -238,7 +238,7 @@ myfile = openfile("private_testfile.txt", True) myfile.writeat("INV", 0) if "private_testfile.txt" in listfiles(): - log("ERROR! File not invisible. + log("ERROR! File not invisible. n") # Write into the file again @@ -252,16 +252,16 @@ try: except RepyArgumentError: pass # There should be an exception here! else: - log("ERROR! Allowed me to delete the file. + log("ERROR! Allowed me to delete the file. n") ``` **Note:** All attacks should be written as Repy V2 files, using the .r2py extension. - -#### Choice of File Names + +#### Choice of File Names ---- It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: @@ -273,9 +273,9 @@ myfile=openfile("look.txt",True) look.txt is a valid file name, however Look.txt is not. Examples of other invalid files names are, look@.txt, look/.txt, and look().txt. Essentially all non-alphanumeric characters are not allowed. - -### Running your security layer + +### Running your security layer ---- Finally, type the following commands at the terminal to run your security layer with your attack program @@ -284,14 +284,14 @@ Finally, type the following commands at the terminal to run your security layer Make sure you went through the "How to get RepyV2" section! - -## Notes and Resources + +## Notes and Resources ---- * For a complete list of syntax in Repyv2 please visit: **[wiki:RepyV2API]** - * The following link is an excellent source for information about security layers: **[http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf]** + * The following link is an excellent source for information about security layers: **[https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf]** * It is possible to add multiple security layers to Repy, this may be useful for testing different mitigations separately. This is done with the following command at the terminal: @@ -302,21 +302,21 @@ Make sure you went through the "How to get RepyV2" section! * In repy log replaces print from python. This may be helpful when testing if Repy installed correctly. - -## Extra Credit + +## Extra Credit ---- For extra credit, make a second security layer which will prevent writing to a private_ file when the first and last three characters in the file **match** and are either "INV" or "PER". If the first three characters in a private_ file are "INV" and the last three are "PER", you should be able to write to the file. Note: Do not submit this code inside your assignment file. Submit a separate copy for extra credit. - -## What to turn in? + +## What to turn in? ---- * Turn in a repy file called reference_monitor_[netid].r2py with all letters in lowercase. * **Never raise unexpected errors or produce any output.** Your program must produce no output when run normally. - * For extra credit turn in a second repy file called extra_credit_[netid].r2py **You must turn in separate files for the normal assignment and extra credit** \ No newline at end of file + * For extra credit turn in a second repy file called extra_credit_[netid].r2py **You must turn in separate files for the normal assignment and extra credit** diff --git a/EducationalAssignments/ProtectFilePartTwo.md b/EducationalAssignments/ProtectFilePartTwo.md index c53155cad..7a105a1b7 100644 --- a/EducationalAssignments/ProtectFilePartTwo.md +++ b/EducationalAssignments/ProtectFilePartTwo.md @@ -1,11 +1,11 @@ -# Security Layer Testing and Penetration +# Security Layer Testing and Penetration In this assignment you will learn how to attack a reference monitor. The reference monitor you will be testing uses the security layer framework (encasement library, etc.) of the Seattle Testbed. It is possible to do this assignment separately however it is recommended that this assignment be completed after [wiki:EducationalAssignments/ProtectFilePartOne ProtectFilePartOne]. Either way you should already have a working security layer or access to one. Testing the security layer is done by running a series of test cases that an adversary may use to circumvent your system. - -## Overview + +## Overview ---- In this assignment you are a tester. You have been given a bunch of reference monitors that need testing before they are deployed. Your job will be to test the security layer with different test cases that try to circumvent the restrictions set by the security layer. If you are able to do so, then the security layer is not secure. The future of the system depends on your ability to test code thoroughly! @@ -16,28 +16,28 @@ Your test cases should test the following design paradigms at work in Part 1 of You will gain points for every student's reference monitor you find a flaw in. It is okay if multiple tests of yours break a student's reference monitor, but you will not gain additional points. - -## Prerequisites + +## Prerequisites ---- This assignment assumes you have Python 2.5, 2.6, or 2.7 and RepyV2 installed on your computer. If you don't already have them please go to [wiki:EducationalAssignments/ProtectFilePartOne#GettingPython GettingPython] and [wiki:EducationalAssignments/ProtectFilePartOne#GettingRepyV2 GettingRepyV2] for a tutorial on how to get them. - -## Helpful links + +## Helpful links ---- The following links will help you become comfortable with Python, Repy and Seattle: * Python tutorial: **[https://docs.python.org/2.7/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * List of RepyV2 API calls: **[wiki:RepyV2API]** - -## Testing the security layers + +## Testing the security layers ---- Test cases are briefly described in [wiki:RepyV2SecurityLayers]. Below is an example of a test case you may want to consider. This test case gives the right 'style' for your all your test cases, but lacks in the number of test cases. A good attack will include many test cases. -### Sample test case +### Sample test case ``` if "private_testfile.txt" in listfiles(): removefile("private_testfile.txt") @@ -49,7 +49,7 @@ myfile = openfile("private_testfile.txt", True) myfile.writeat("INV", 0) if "private_testfile.txt" in listfiles(): - log("ERROR! File not invisible. + log("ERROR! File not invisible. n") # Write into the file again @@ -63,21 +63,21 @@ try: except RepyArgumentError: pass # There should be an exception here! else: - log("ERROR! Did not raise an exception when it was supposed to. + log("ERROR! Did not raise an exception when it was supposed to. n") ``` -### Code analysis +### Code analysis In this code we first write "INV" into a file (at offset 0) and then check the list returned by the listfiles() method. If private_testfile.txt is in that list, we log that as an error since the file was supposed to be hidden from the list. Next, we write "PER" into the file (again, at offset 0) and try to delete that file. This should raise an exception if the security layer is working correctly. If no exception is raised, then we log this as an error. -### More information on: Try, Except, Else, Finally +### More information on: Try, Except, Else, Finally The try, except, else and finally statements are a part of **exception handling**. For more information on exception handling please visit: * [http://docs.python.org/tutorial/errors.html] * [http://wiki.python.org/moin/HandlingExceptions] * [http://www.tutorialspoint.com/python/python_exceptions.htm] -### Hints and ideas for testing +### Hints and ideas for testing When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: * threading @@ -85,28 +85,28 @@ When writing your own tests it is important to test for a complete set of possib And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. - -## Notes and Resources + +## Notes and Resources ---- - * The following link is an excellent source for information about security layers: http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf * In repy 'log' replaces 'print' from python. Many students find this to be a stumbling block. - -## Important Note + +## Important Note ---- Your test case should not output anything if the security layer passes it. It should print appropriate error message only if the security layer fails. - -## Extra Credit + +## Extra Credit ---- Find bugs in the extra credit reference monitors given the altered threat model. You should include more test cases in the extra credit! - -## How to run your tests on many reference monitors + +## How to run your tests on many reference monitors ---- If you are using Mac or Linux, you can do something like the following: @@ -117,9 +117,9 @@ Then you can type the following in the bash shell to execute the test cases with for referencemonitor in reference_*; do for testcase in netid_*.r2py; do python repy.py restrictions.default encasementlib.r2py $referencemonitor $testcase; done; done ``` - -## What to turn in? + +## What to turn in? ---- * Turn in the test cases used to attack the reference monitors in a zip file. The name of each testcase must start with your Net ID in lowercase. For example: ab321_securitytest1.r2py. - * Optionally turn in the test cases used to attack the extra credit reference monitors in a separate zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: ab321_run_twice_metadata_removal.r2py, ab321_run_once_threading_hack.r2py. \ No newline at end of file + * Optionally turn in the test cases used to attack the extra credit reference monitors in a separate zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: ab321_run_twice_metadata_removal.r2py, ab321_run_once_threading_hack.r2py. diff --git a/EducationalAssignments/SecurityLayerPartOne.md b/EducationalAssignments/SecurityLayerPartOne.md index f5707827b..b1814ad63 100644 --- a/EducationalAssignments/SecurityLayerPartOne.md +++ b/EducationalAssignments/SecurityLayerPartOne.md @@ -1,5 +1,5 @@ -# Building a Security Layer +# Building a Security Layer This assignment will help you understand security mechanisms. You will be guided through the steps of creating a reference monitor using the security layer functionality in Repy V2. A reference monitor is an access control concept that refers to an abstract machine @@ -12,9 +12,9 @@ This assignment is intended to prepare you for thinking about security paradigms - -## Overview + +## Overview ---- In this assignment an attacker attempts to write "MZ" to the first two characters of a file. (Historically, this is an indicator for an [old executable format](http://www.fileformat.info/format/exe/corion-mz.htm).) Your goal is to prevent the attacker from succeeding. Thus you must stop a program from creating a file with "MZ" as the first two characters. You will do this by adding security rules to the functions available for reading from a file and writing to a file. The future of the system depends on your ability to write secure code! @@ -25,9 +25,9 @@ Three design paradigms are at work in this assignment: accuracy, efficiency, and * Efficiency: The security layer should use a minimum number of resources, so performance is not compromised. This means you should not do things like re-read the file before each write. * Security: The attacker should not be able to circumvent the security layer. Hence, if "MZ" can be written two the first two characters, the security is compromised. - -### Getting Python + +### Getting Python ---- Please note you must have Python 2.6 or 2.7 to complete this assignment. Instructions on how to get Python for Windows can be found [InstallPythonOnWindows here]. If you are using Linux or a Mac it is likely you already have Python. In order to verify this, simply open a terminal and type ```python```. Please check its version on the initial prompt. @@ -35,7 +35,7 @@ Please note you must have Python 2.6 or 2.7 to complete this assignment. Instruc -### Getting RepyV2 +### Getting RepyV2 ---- The preferred way to get Repy is ''installing from source''. For this, you check out the required Git repositories, and run a build script. ([wiki:BuildInstructions This page] describes the process in greater detail). You can always update the repos later, and rebuild, so that you get the latest stable version of the Repy runtime. @@ -84,7 +84,7 @@ In order to test whether or not these steps worked, please copy and paste the co If you got an error, please go through the troubleshooting section found below. -### Troubleshooting Repy code +### Troubleshooting Repy code If you can't get Repy files to run, some of the following common errors may have occurred: * using `print` instead of `log`: @@ -108,25 +108,25 @@ To run the unit test, which will automatically tell you if you have errors with * [wiki:RepyV2CheckoutAndUnitTests] --> - -### Tutorials for Repy and Python + +### Tutorials for Repy and Python ---- Now that you have Repy and Python, you may need a refresher on how to use them. The following tutorials are excellent sources of information. * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Building the security layer + +## Building the security layer ---- **[wiki:RepyV2SecurityLayers]** explains the syntax used to build a reference monitor. The general tutorials above will aid in looking up other details about Repy. Remember, you have no idea how the attacker will try to penetrate your security layer, so it is important that you leave nothing to chance! Your reference monitor should try to stop every attack you can think of. Not just one or two. A good reference monitor will do this and incorporate the design paradigms of accuracy, efficiency and security defined above. - -## A basic (and inadequate) defense + +## A basic (and inadequate) defense Time to start coding! Let's inspect a basic security layer. @@ -199,25 +199,25 @@ CHILD_CONTEXT_DEF["openfile"] = {TYPE:OBJC,ARGS:(str,bool),EXCP:Exception,RETURN secure_dispatch_module() ``` -## Using the example layer +## Using the example layer Keep in mind the above security layer would only stop one kind of attack. Thus if an attacker doesn't know much about file input/output this will probably stop them. However there are a bunch of tricks that can be used in order to circumvent this security layer easily. For instance, the above reference monitor isn't thread safe. For an introduction to thread safety please read [wiki/Thread_safety](http://en.wikipedia.org/wiki/Thread_safety). - -### Code analysis + +### Code analysis The `writeat()` function attempts follow the design principles of accuracy, efficiency, and security. However it does so inadequately. Accuracy is attempted by only blocking "MZ" from being written. If any other characters are written, the `writeat()` function executes normally. Efficiency is preserved because the number of test cases is small. Also security is attempted because "MZ" cannot be written to a file as easily. It is important to note that no security system is perfect. It is more important to make a security system that is infeasible to break, rather than impossible to break. However this security layer does not yet create the necessary infeasibility requirements for most attacks. Thus we see that there is a grey area of what is an acceptable level of impedance. Now let's look at the structure of `writeat()`. First off, the if-statement contains the mitigation. If the "alarm" is not tripped then the `writeat()` function is executed normally. Within the if-statement `data.startswith("MZ")` is used to check for an illegal write. Specifically, `data` is a variable used to store information which will be written to the file. And the method `startswith()` checks to see if the file begins with certain characters, in this case "MZ". The second condition, `(offset == 0)` determines the starting point of the cursor before it writes to the file. Thus offset = 0, means we start writing at the beginning of the file. Similarly if offset =4, the computer would start writing to the 5th position of the file. Starting with 0 rather than 1 has heuristic meaning in computer science. - -## Testing your security layer + +## Testing your security layer ---- In this part of the assignment you will pretend to be an attacker. Remember the attacker's objective is to write "MZ" to a file. By understanding how the attacker thinks, you will be able to write better security layers. Perhaps while attacking your security layer you will think of a new mitigation that should have been implemented. Keep in mind attacks are attempts to mitigate a given security protocol. If even one case succeeds, then your security layer has been compromised. Thus the attack you write should include several methods of attempting to write "MZ" to a file. An example of an attack is found below: @@ -231,13 +231,12 @@ try: # It raised an Exception (as it was supposed to): except ValueError: - log("The security layer correctly blocked the write!!! + log("The security layer correctly blocked the write!!! n") # No Exception was raise else: - log("Wrote an MZ!!! -n") + log("Wrote an MZ!!! \n") finally: # Close the file after our attempt. @@ -246,7 +245,7 @@ finally: **Note:** All attacks should be written as Repy files, using the .r2py extension. -### Code Analysis +### Code Analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -259,9 +258,9 @@ look.txt is a valid file name, however Look.txt is not. Examples of other inval This code attempts to write "MZ" to the file directly. First the file is opened using `myfile=openfile("look.txt",True)`. Next `myfile.writeat("MZ",0)` tries to write "MZ" to the file. The 0 refers to an offset of zero. The `try:` statement tells the program to "try" this case. Notice that the `except` is executed if an error is raised. If the security layer fails the test then the else statement is executed. The `finally:` statement will always run, closing the file. - -### Running your security layer + +### Running your security layer ---- Finally, type the following commands at the terminal to run your security layer with your attack program @@ -269,10 +268,10 @@ Finally, type the following commands at the terminal to run your security layer '''Make sure you use filenames that are all lowercase letters! - -# Notes and Resources + +# Notes and Resources ---- * A list of command line utilities for Windows can be found at **[http://commandwindows.com/command3.htm]** @@ -297,18 +296,18 @@ Finally, type the following commands at the terminal to run your security layer * In repy log replaces print from python. This may be helpful when testing if Repy installed correctly. - -# Extra Credit + +# Extra Credit ---- For extra credit, make a second security layer which stops an attacker from writing 'p0wnd' anywhere in a file. This security layer will be similar to the first one you wrote, except now you must handle the case of `offset != 0` as well as a few other things. You should also write an attack layer which attempts to write 'p0wnd' to a file. **Hint:** Since this string can be written anywhere in the file, there will most likely be more test cases. - -# What to turn in? + +# What to turn in? ---- * Turn in a repy file called reference_monitor_[ name or id number].r2py - * For extra credit turn in a second repy file called extra_credit_[ name or id number].r2py \ No newline at end of file + * For extra credit turn in a second repy file called extra_credit_[ name or id number].r2py diff --git a/EducationalAssignments/SecurityLayerPartTwo.md b/EducationalAssignments/SecurityLayerPartTwo.md index 3e8c730fb..92ec63e13 100644 --- a/EducationalAssignments/SecurityLayerPartTwo.md +++ b/EducationalAssignments/SecurityLayerPartTwo.md @@ -1,12 +1,12 @@ -# Security layer testing and penetration +# Security layer testing and penetration In this assignment you will learn how to attack a reference monitor. The reference monitor you will be testing uses the security layer framework (encasement library, etc.) for the Seattle testbed. It is possible to do this assignment separately however it is recommended that this assignment be you completed after [wiki:EducationalAssignments/SecurityLayerPartOne SecurityLayerPartOne]. Either way you should already have a working security layer or access to one. Testing the security layer is done by running a series of test cases that an adversary may use to circumvent your system. This assignment is intended to prepare you for thinking about security paradigms in a functional way. The ideas of information, security and privacy have been embedded into the steps of this assignment. - -## Overview + +## Overview ---- In this assignment you are a tester. You have been sent a bunch of reference monitors that need testing before they are deployed. Your job will be to ensure an attacker cannot circumvent these security layers. In order to do this, you will attempt to write "MZ" to the first two characters of a file. If you are able to do so, then the security layer is not secure. The future of the system depends on your ability to test code thoroughly! @@ -31,24 +31,24 @@ After testing, write a report on your findings, including all attacks that succe Note: For a full description on MZ executable files see http://www.fileformat.info/format/exe/corion-mz.htm - -## Prerequisites + +## Prerequisites This assignment assumes you have Python2.5 or Python2.6, Repy and RepyV2 installed on your computer. If you don't already have them please go to [wiki:EducationalAssignments/SecurityLayerPartOne#GettingPythonRepy SecurityLayerPartOne] for a tutorial on how to get them. - -### Helpful links + +### Helpful links ---- The following links will aid students in becoming comfortable with Python, Repy and seattle: * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Testing security layers + +## Testing security layers ---- -### Hypothesis, test case, counter example +### Hypothesis, test case, counter example The goal of a good tester is to test hypotheses. A hypothesis is just a scientific way of asking a question. The hypothesis of this assignment is "This security layer is well designed." The questions you will ask when running your test cases will always be the same @@ -62,11 +62,11 @@ Notice that these questions are parallels of the security paradigms: security, e If we can find a case where the hypothesis is false, then the security layer is not secure. Such a case is referred to as a counter example. Hence all test cases should be designed to test for these three types of flaws. - -### Examples of tests + +### Examples of tests Test cases are briefly described in [wiki:EducationalAssignments/SecurityLayerPartOne SecurityLayerPartOne] and [wiki:RepyV2SecurityLayers]. Below is another example of a test case you may want to consider. This test case gives the right 'style' for your all your test cases, but lacks in the number of test cases. A good attack will include many test cases. -#### Test case 1: +#### Test case 1: ``` # Open a file @@ -88,7 +88,7 @@ finally: # Close the file after our attempt. myfile.close() ``` -#### Code analysis +#### Code analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -101,7 +101,7 @@ look.txt is a valid file name, however Look.txt is not. Examples of other inval In this case we are verifying the security of the reference monitor. This code attempts to write "MZ" to the file directly. First the file is opened using myfile=openfile("look.txt",True). Next myfile.writeat("MZ",0) tries to write "MZ" to the file. The 0 refers to an offset of zero. The try: statement tells the program to "try" this case. Notice that the except is executed if an error is raised. If the security layer fails the test then the else statement is executed. The finally: statement will always run, closing the file. -#### Test case 2: +#### Test case 2: ``` # Open a file @@ -123,20 +123,20 @@ finally: # Close the file after our attempt. myfile.close() ``` -#### Code analysis +#### Code analysis In this case we are verifying the accuracy of the reference monitor. This code attempts to write "MZ" to the file directly. First the file is opened using myfile=openfile("look.txt",True). Next myfile.writeat("MZ",1) tries to write "MZ" to the file. The 1 refers to the fact that the characters are placed as the second and third characters in the file. The try: statement tells the program to "try" this case. Notice that the except is executed if an error is raised. If the security layer fails the test then the else statement is executed. The finally: statement will always run, closing the file. If this case produces anything other than "Write okay", then this layer fails the accuracy design paradigm. The security layer should only stop a file from starting with "MZ", not containing "MZ". -#### More information on: Try, Except, Else, Finally +#### More information on: Try, Except, Else, Finally The try, except, else and finally statements are part of **exception handling**. For more information on exception handling please visit: * [http://docs.python.org/tutorial/errors.html] * [http://wiki.python.org/moin/HandlingExceptions] * [http://www.tutorialspoint.com/python/python_exceptions.htm] -### Hints and Ideas for testing +### Hints and Ideas for testing When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: @@ -145,12 +145,12 @@ When writing your own tests it is important to test for a complete set of possib * writing to both positions at once And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. - -## Notes and Resources + +## Notes and Resources ---- - * The following link is an excellent source for information about security layers: http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf * [repy_v2/benchmarking-support/allnoopsec.py](https://seattle.poly.edu/browser/seattle/branches/repy_v2/benchmarking-support/allnoopsec.py) is an empty security layer that doesn't perform any operations. @@ -159,16 +159,16 @@ And more! Remember a good security layer can't be broken by anyone! Which is a * In repy 'log' replaces 'print' from python. Many students find this to be a stumbling block. - -## Extra Credit + +## Extra Credit ---- Write and test a second reference monitor that stops "p0wnd" from being written anywhere in a file. You should include more test cases in the extra credit! - -## What to turn in? + +## What to turn in? ---- * Turn in the test cases used to attack a given reference monitor. * Turn in the write up of each reference monitor attacked, with critique of the reference monitor. Critique should include all test cases that worked. - * Write up for 'p0wnd' reference monitors. \ No newline at end of file + * Write up for 'p0wnd' reference monitors. diff --git a/EducationalAssignments/SetMaxFileSizePartOne.md b/EducationalAssignments/SetMaxFileSizePartOne.md index 9a51c08d2..65a204d37 100644 --- a/EducationalAssignments/SetMaxFileSizePartOne.md +++ b/EducationalAssignments/SetMaxFileSizePartOne.md @@ -1,4 +1,4 @@ -# Implement a Defensive Security System +# Implement a Defensive Security System This assignment will help you understand security mechanisms. You will be guided through the steps of creating a reference monitor using the security layer functionality in Repy V2. A reference monitor is an access control concept that refers to an abstract machine that mediates all access to objects by subjects. This can be used to allow, deny, or change the behavior of any set of calls. While not a perfect way of validating your reference monitor, it is useful to create test cases to see whether your security layer will work as expected. (The test cases may be turned in as part of the next assignment.) @@ -7,9 +7,9 @@ This assignment is intended to reinforce concepts about access control and refer - -## Overview + +## Overview ---- In this assignment you will create a security layer which stops the user from writing to a file after a specified file size. You will add a function called setmaxfilesize() that will take a file size as parameter and enforce it for the specified file. If a file is longer than the specified size, any data past this point in the file will be truncated. (Since the Repy V2 API does not have a truncate call, you will have to implement this functionality yourself.) If a writeat() call is performed at a location at or past the end of the file size limit, a SeekPastEndOfFileError should be raised. If the writeat() starts before the file limit, any data up to the size limit should be written and all other data should be discarded. @@ -22,15 +22,15 @@ Three design paradigms are at work in this assignment: accuracy, efficiency, and * Security: The attacker should not be able to circumvent the security layer. Hence, if the data can be written beyond specified size, the security is compromised. - -### Getting Python + +### Getting Python ---- Please note you must have Python 2.5, 2.6, or 2.7 to complete this assignment. Instructions on how to get Python for Windows can be found [InstallPythonOnWindows here]. If you are using Linux or a Mac it is likely you already have Python. In order to verify this, simply open a terminal and type ```python```. Please check its version on the initial prompt. **Note:**If you are using Windows, you will need python in your path variables. A tutorial on adding variables to your path in Windows can be found at [http://www.computerhope.com/issues/ch000549.htm] -### Getting RepyV2 +### Getting RepyV2 ---- The preferred way to get Repy is ''installing from source''. For this, you check out the required Git repositories, and run a build script. You can always update the repos later, and rebuild, so that you get the latest stable version of the Repy runtime. @@ -71,7 +71,7 @@ In order to test whether or not these steps worked, please copy and paste the co If you got an error, please go through the trouble shooting section found below. -### Troubleshooting Repy code +### Troubleshooting Repy code ---- If you can't get Repy files to run, some of the following common errors may have occurred: @@ -96,24 +96,24 @@ To run the unit test, which will automatically tell you if you have errors with * [wiki:RepyV2CheckoutAndUnitTests] --> - -### Tutorials for Repy and Python + +### Tutorials for Repy and Python ---- Now that you have Repy and Python, you may need a refresher on how to use them. The following tutorials are excellent sources of information. * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 API calls: **[wiki:RepyV2API]** - -## Building the security layer + +## Building the security layer ---- The following program is a basic and incomplete sample code for you to get an idea about writing security layer. Remember, you have no idea how the attacker will try to penetrate your security layer, so it is important that you leave nothing to chance! - -### A basic (and inadequate) defense + +### A basic (and inadequate) defense Time to start coding! Let's inspect a basic security layer. @@ -196,9 +196,9 @@ CHILD_CONTEXT_DEF["removefile"]["target"] = secure_removefile secure_dispatch_module() ``` - -### Testing your security layer + +### Testing your security layer ---- In this part of the assignment you will pretend to be an attacker. Remember the attacker's objective is to bypass the file size restrictions or cause the security layer to act in a disallowed manner. By understanding how the attacker thinks, you will be able to write better security layers. @@ -220,7 +220,7 @@ try: except SeekPastEndOfFileError: pass # there should be an exception here... else: - log("ERROR! Didn't stop me from writing past the maximum file size. + log("ERROR! Didn't stop me from writing past the maximum file size. n") @@ -230,7 +230,7 @@ myfile.close() **Note:** All attacks should be written as Repy V2 files, using the .r2py extension. -#### Choice of File Names +#### Choice of File Names ---- It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: @@ -241,21 +241,21 @@ myfile=openfile("look.txt",True) look.txt is a valid file name, however Look.txt is not. Examples of other invalid files names are, look@.txt, look/.txt, and look().txt. Essentially all non-alphanumeric characters are not allowed. -### Running your security layer +### Running your security layer ---- Finally, type the following commands at the terminal to run your security layer with your attack program ```python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py ``` Make sure you went through the "How to get RepyV2" section! - -# Notes and Resources + +# Notes and Resources ---- * For a complete list of syntax in Repyv2 please visit: **[wiki:RepyV2API]** - * The following link is an excellent source for information about security layers: **[http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf]** + * The following link is an excellent source for information about security layers: **[https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf]** * **Note:** It is possible to add multiple security layers to Repy, this may be useful for testing different mitigations separately. This is done with the following command at the terminal: @@ -264,17 +264,17 @@ Make sure you went through the "How to get RepyV2" section! **Your security layer should produce no output!! ** * In repy log replaces print from python. This may be helpful when testing if Repy installed correctly. - -# Extra Credit + +# Extra Credit ---- For extra credit, make a second security layer which retains its specified size limits even if it is closed and reopened again. The file size limit should stay after the program is terminated and restarted Do not submit this code inside your assignment. Submit a separate copy for extra credit. - -# What to turn in? + +# What to turn in? ---- * Turn in a repy file called reference_monitor_[ netid ].r2py with all letters in lowercase. * **Never raise unexpected errors or produce any output.** Your program must produce no output when run normally. - * For extra credit turn in a second repy file called extra_credit_[netid].r2py **You must turn in separate files for the normal assignment and extra credit** \ No newline at end of file + * For extra credit turn in a second repy file called extra_credit_[netid].r2py **You must turn in separate files for the normal assignment and extra credit** diff --git a/EducationalAssignments/SetMaxFileSizePartTwo.md b/EducationalAssignments/SetMaxFileSizePartTwo.md index 3e0b6f221..babbc62ba 100644 --- a/EducationalAssignments/SetMaxFileSizePartTwo.md +++ b/EducationalAssignments/SetMaxFileSizePartTwo.md @@ -1,12 +1,12 @@ -# Implement a Security Attack and Penetration Test System +# Implement a Security Attack and Penetration Test System In this assignment you will learn how to attack a reference monitor. The reference monitor you will be testing uses the security layer framework (encasement library, etc.) for the Seattle testbed. It is possible to do this assignment separately however it is recommended that this assignment be completed after [wiki:EducationalAssignments/SetMaxFileSizePartOne SetMaxFileSizePartOne]. Either way you should already have a working security layer or access to one. Testing the security layer is done by running a series of test cases that an adversary may use to circumvent your system. - -## Overview + +## Overview ---- In this assignment you are a tester. You have been sent a bunch of reference monitors that need testing before they are deployed. Your job will be to test security layer with different test cases that try to exceed the file size beyond permission. If you are able to do so, then the security layer is not secure. The future of the system depends on your ability to test code thoroughly! @@ -20,26 +20,26 @@ Three design paradigms are at work in this assignment: accuracy, efficiency, and You will submit a zip file containing all of the tests you have created. You will gain points for every student's reference monitor you find a flaw in. It is okay if multiple tests of yours breaking a student's reference monitor, but you will not gain additional points. - -## Prerequisites + +## Prerequisites ---- This assignment assumes you have Python2.5 or Python2.6, Repy and RepyV2 installed on your computer. If you don't already have them please go to [wiki:EducationalAssignments/SetMaxFileSizePartOne#GettingPythonRepy SetMaxFileSizePartOne] for a tutorial on how to get them. - -### Helpful links + +### Helpful links ---- The following links will aid students in becoming comfortable with Python, Repy and seattle: * Python tutorial: **[http://docs.python.org/tutorial/]** * Seattle tutorial: **[https://seattle.poly.edu/wiki/PythonVsRepy]** * list of RepyV2 syntax: **[wiki:RepyV2API]** - -## Testing security layers + +## Testing security layers ---- -### Examples of tests +### Examples of tests Test cases are briefly described in [wiki:EducationalAssignments/SetMaxFileSizePartOne SetMaxFileSizePartOne] and [wiki:RepyV2SecurityLayers]. Below is another example of a test case you may want to consider. This test case gives the right 'style' for your all your test cases, but lacks in the number of test cases. A good attack will include many test cases. -#### Test case 1: +#### Test case 1: ``` if "look.txt" in listfiles(): @@ -56,7 +56,7 @@ myfile.setmaxfilesize(10) try: myfile.writeat("Testing",2) except SeekPastEndOfFileError: - log("Security layer failed because it blocked a valid write operation + log("Security layer failed because it blocked a valid write operation n") else: #user is allowed to write if its at valid place @@ -66,7 +66,7 @@ else: #Close the file myfile.close() ``` -#### Code analysis +#### Code analysis It is important to keep in mind that only lowercase file names are allowed. So in the above code, specifically: ``` @@ -79,7 +79,7 @@ look.txt is a valid file name, however Look.txt is not. Examples of other inval In this code first we write something in file and then set the size of file to 10 using setmaxfilesize() method. Then we try to write again within valid offset and file size range. If security layer doesn't allow us to do and raises error for a valid write operation that it is failed. -#### Test case 2: +#### Test case 2: ``` if "look.txt" in listfiles(): @@ -107,14 +107,14 @@ else: myfile.close() ``` -#### More information on: Try, Except, Else, Finally +#### More information on: Try, Except, Else, Finally The try, except, else and finally statements are part of **exception handling**. For more information on exception handling please visit: * [http://docs.python.org/tutorial/errors.html] * [http://wiki.python.org/moin/HandlingExceptions] * [http://www.tutorialspoint.com/python/python_exceptions.htm] -### Hints and Ideas for testing +### Hints and Ideas for testing When writing your own tests it is important to test for a complete set of possible penetrations. Keep in mind, it only takes one test case to break through a security layer. Some of the things you may want to test for include: @@ -122,29 +122,29 @@ When writing your own tests it is important to test for a complete set of possib * writing to multiple files And more! Remember a good security layer can't be broken by anyone! Which is all part of the fun! It's about solving a puzzle. First you make the puzzle - write the security layer, then you solve the puzzle - try to bypass it. If your puzzle is "good enough", no one will be able to break it, no matter what. - -## Notes and Resources + +## Notes and Resources ---- - * The following link is an excellent source for information about security layers: http://isis.poly.edu/~jcappos/papers/cappos_seattle_ccs_10.pdf + * The following link is an excellent source for information about security layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf * In repy 'log' replaces 'print' from python. Many students find this to be a stumbling block. -## Important Note +## Important Note ---- Your Test Case should not output anything if the security layer passes it. It should print appropriate error message only if the security layer fails. - -## Extra Credit + +## Extra Credit ---- Find bugs in the extra credit reference monitors given the altered threat model. You should include more test cases in the extra credit! - -## How to run your tests on many reference monitors + +## How to run your tests on many reference monitors ---- If you are using Mac or Linux, you can do something like the following: @@ -159,10 +159,10 @@ for referencemonitor in reference_*; do for testcase in netid_*.r2py; do python - -## What to turn in? + +## What to turn in? ---- * Turn in the test cases used to attack a given reference monitor in a zip file. The name of each testcase must start with your net id in lowercase. For example: us341_securitytest1.r2py justincappos_goodaccuracytest.r2py are both valid names. - * Optionally turn in the test cases used to attack the extra credit reference monitors in a zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: jcappos_run_twice_metadata_removal.r2py jcappos_run_once_threading_hack.r2py. \ No newline at end of file + * Optionally turn in the test cases used to attack the extra credit reference monitors in a zip file. Note that in this case, you can expect that your code is run more than once. In the name of the file, say if it needs to be run multiple times. For example: jcappos_run_twice_metadata_removal.r2py jcappos_run_once_threading_hack.r2py. diff --git a/EducationalAssignments/UndoPartOne.md b/EducationalAssignments/UndoPartOne.md new file mode 100644 index 000000000..358089b7f --- /dev/null +++ b/EducationalAssignments/UndoPartOne.md @@ -0,0 +1,304 @@ +# Implement a Defensive Security System + +This assignment will help you understand security mechanisms. You will be guided +through the steps of creating a reference monitor using the security layer +functionality in Repy V2. A reference monitor is an access control concept that +refers to an abstract machine that mediates all access to objects by subjects. +This can be used to allow, deny, or change the behavior of any set of calls. +While not a perfect way of validating your reference monitor, it is useful to +create test cases to see whether your security layer will work as expected (the +test cases may be turned in as part of the next assignment). + +This assignment is intended to reinforce concepts about access control and +reference monitors in a hands-on manner. + + +## Overview + +In this assignment, you will implement a defense monitor to oversee file +operations in Repy. The monitor will enhance the default Repy behavior by adding +an undo functionality for writes. This defense monitor will ensure that software +follows certain rules and guidelines, preventing potential unauthorized actions. + +The focus is on monitoring file write operations and allowing the last write to +be reverted. This undo capability mimics common document editors that let you +reverse your most recent edit. + +You should write test applications to ensure your reference monitor behaves +properly in different cases and to test attacks against your monitor. + + +### Specifications: + +1. Your defense monitor should incorporate all the standard file operation + methods, with an added method named `undo`. +2. A `writeat` operation will not immediately write its changes to the file. The + changes will only be permanent (commit) after either of the following occur: + - A subsequent valid `writeat` operation is executed. + - The file is closed. +3. The `undo` operation will reverse the changes of the last valid `writeat` + operation, making it as if it didn't happen. It's crucial to note that only + the most recent operation can be undone. If there haven't been any `writeat` + operations, the `undo` has no effect. If several `writeat` operations are + consecutively executed, only the changes from the last valid `writeat` can be + undone. +4. Aside from the ability to be undone and potentially delayed writes, all + `writeat` operations should behave the same way as they do in the RepyV2 API. +5. The `undo` operation raises `FileClosedError` if the file is already closed. +6. `readat` cannot read data that has not been committed to the file. + +Three design paradigms are at work in this assignment: accuracy, efficiency, and +security. + + * Accuracy: The defense monitor should precisely and consistently manage file +operations. Only specific operations, such as `writeat`, should have their +behavior modified. All situations that are not described above *must* match that +of the underlying API. + + * Efficiency: The security layer should use a minimum number of resources, so +performance is not compromised. For example, you may not call `readat()` +everytime `writeat()` is called. It is permissable to call `readat()` upon +`fileopen()`, however. + + * Security: The defense layer should be robust against tampering and +circumvention. Attackers must not be able to bypass, disable, or exploit the +defense monitor's enhanced behaviors, ensuring the integrity and intended +functionality of file operations. + + +### Getting Python and RepyV2 + +Please refer to the [SeattleTestbed Build +Instructions](../Contributing/BuildInstructions.md#prerequisites) for details. + +Once you have built RepyV2 into a directory of your choice, change into that +directory. Use the command below in order to run your RepyV2 applications: + +```bash +python2 repy.py restrictions.default encasementlib.r2py [security_layer].r2py [application].r2py +``` + +(Replace `[security_layer].r2py` and `[application].r2py` by the names of the +security layers and application that you want to run.) + +In order to test whether or not these steps worked, please copy and paste the +code found below for the sample security layer and sample attack. + +You should get an `RepyArgumentError` when running the test. If not, please go +through the troubleshooting section found below. + + +### Troubleshooting Repy code + +If you can't get Repy files to run, some of the following common errors may have +occurred: + + * using `print` instead of `log`: + +Repy is a subset of Python, but its syntax is slightly different. For example, +Python's `print` statement cannot be used; Repy has `log` for that. For a full +list of acceptable syntax please see +[https://github.com/SeattleTestbed/docs/blob/master/Programming/RepyV2API.md] + + * command line errors: + +**files are missing:** In the above command line call, you must have `repy.py`, +restrictions.default, encasementlib.r2py, the security layer and the program you +want to run in the current working directory. If any or all of the above files +are not in that directory then you will not be able to run repy files. + + + + +### Tutorials for Repy and Python + +Now that you have Repy and Python, you may need a refresher on how to use them. +The following tutorials provide this information. + + * Official [Python tutorial](http://docs.python.org/tutorial/) + * [Differences between RepyV2 and Python](../Programming/PythonVsRepyV2.md) + * List of [RepyV2 API calls](../Programming/RepyV2API.md) + + +## Building the security layer + +The following program is a sample security layer, it is not complete and does +not handle all cases required by the API. Remember, you have no idea how the +attacker will try to penetrate your security layer, so it is important that you +leave nothing to chance! + + +### A basic (and inadequate) defense + +Time to start coding! Let's inspect a basic security layer. + +```py +""" +This security layer inadequately handles the undo functionality + +Note: + This security layer uses encasementlib.r2py, restrictions.default, repy.py and Python + Also you need to give it an application to run. + python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py + +""" +TYPE = "type" +ARGS = "args" +RETURN = "return" +EXCP = "exceptions" +TARGET = "target" +FUNC = "func" +OBJC = "objc" + + +class LPFile(): + def __init__(self, filename, create): + # globals + mycontext['debug'] = False + self.LPfile = openfile(filename, create) + self.pending_data = None + self.pending_offset = None + + def readat(self, bytes, offset): + # Read from the file using the sandbox's readat... + return self.LPfile.readat(bytes, offset) + + def writeat(self, data, offset): + self.LPfile.writeat(self.pending_data, self.pending_offset) + self.pending_data = data + self.pending_offset = offset + + def undo(self): + self.pending_data = None + self.pending_offset = None + + def close(self): + self.LPfile.close() + +def LPopenfile(filename, create): + return LPFile(filename, create) + +# The code here sets up type checking and variable hiding for you. +# You should not need to change anything below here. +sec_file_def = { + "obj-type": LPFile, + "name": "LPFile", + "writeat": {"type": "func", "args": (str, (int, long)), "exceptions": Exception, "return": (int, type(None)), "target": LPFile.writeat}, + "readat": {"type": "func", "args": ((int, long, type(None)), (int, long)), "exceptions": Exception, "return": str, "target": LPFile.readat}, + "undo": {"type": "func", "args": None, "exceptions": None, "return": type(None), "target": LPFile.undo}, + "close": {"type": "func", "args": None, "exceptions": Exception, "return": (bool, type(None)), "target": LPFile.close} +} + +CHILD_CONTEXT_DEF["openfile"] = { + TYPE: OBJC, + ARGS: (str, bool), + EXCP: Exception, + RETURN: sec_file_def, + TARGET: LPopenfile +} + +# Execute the user code +secure_dispatch_module() +``` + + +### Testing your security layer + +In this part of the assignment you will pretend to be an attacker. Remember the +attacker's objective is to bypass the A/B restrictions or cause the security +layer to act in a disallowed manner. By understanding how the attacker thinks, +you will be able to write better security layers. + +An example of an attack is found below: + +```py +# clean up if the file exists. +if "testfile.txt" in listfiles(): + removefile("testfile.txt") + +# create a file +myfile=openfile("testfile.txt",True) + +# intial write to the file +myfile.writeat("12345678",0) + +# attempt to overwrite the beginning of the file +myfile.writeat("Hi!",0) + +# now, undo the previous write +myfile.undo() + +# the contents should still be "12345678" as the overwrite was undone +assert("12345678" == myfile.readat(8,0)) + +# close the file +myfile.close() +``` + +If the reference monitor is correct, there should be no `RepyArgumentError`. + +**Note:** All attacks should be written as Repy V2 files, using the `.r2py` +extension. + + +### Choice of File Names + +Filenames may only be in the current directory and may only contain lowercase +letters, numbers, the hyphen, underscore, and period characters. Also, filenames +cannot be '.', '..', the blank string or start with a period. There is no +concept of a directory or a folder in repy. Filenames must be no more than 120 +characters long. + + +### Running your security layer + +Finally, type the following commands at the terminal to run your security layer +with your attack program + +```bash +python repy.py restrictions.default encasementlib.r2py [security_layer].r2py [attack_program].r2py +``` + +Make sure you went through the "How to get RepyV2" section! + + +# Notes and Resources + + * For a complete list of syntax in Repyv2 please visit: + **[https://github.com/SeattleTestbed/docs/blob/master/Programming/RepyV2API.md]** + + * The following link is an excellent source for information about security + layers: + **[https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf]** + + * **Note:** It is possible to add multiple security layers to Repy, this may be +useful for testing different mitigations separately. This is done with the +following command at the terminal: + +```bash +python repy.py restrictions.default encasementlib.r2py [security_layer1].r2py [security_layer2].r2py [security_layer3].r2py [program].r2py +``` + + * **Your security layer must produce no output!!** + + * In repy log replaces print from python. This may be helpful when testing if +Repy installed correctly. + + +# What to turn in? + + * Turn in a repy file called `reference_monitor_[ netid ].r2py` with all +letters in lowercase. + +* **Never raise unexpected errors or produce any output.** Your program must +produce no output when run normally. diff --git a/EducationalAssignments/UndoPartThree.md b/EducationalAssignments/UndoPartThree.md new file mode 100644 index 000000000..ff45ce0d5 --- /dev/null +++ b/EducationalAssignments/UndoPartThree.md @@ -0,0 +1,105 @@ +# Fixing your security layer + +In this assignment you will analyze the bugs in your security layer from [Part +One](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/UndoPartOne.md) +and fix them. You may want to use test cases from [Part +Two](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/UndoPartTwo.md) +to help identify these bugs. + + +## Overview + +In this assignment you are fixing your reference monitor. You have been given a +bunch of test programs that try to compromise your reference monitor. Your job +will be to determine where your reference monitor failed, and fix it to ensure +an attacker cannot circumvent the security layer. + + +## Common Bugs + + * The reference monitor needs to track the state of the information on disk, + but cannot re-read it for every access (due to efficiency concerns). A common + mistake is when the attacker can cause the reference monitor’s state to + diverge from the underlying system’s state, especially in error conditions. + + * Time-of-check-to-time-of-use (TOCTTOU) bugs and other types of race + conditions are a fairly common oversight for students. Some students write + test cases that attempt to trigger a race condition to exploit these + problems. This can result in essentially any sort of attack, even infinite + loops in the reference monitor in some cases. + + * Some reference monitors inappropriately share state for different files. For + example, there may be a global set of state that is used to store the first + two bytes. By opening multiple files, an attacker may be able to overwrite + this state and cause security and accuracy issues. + + * In rare cases, a student’s reference monitor may inappropriately retain state + for a file. For example, an attacker may create a file, write some data, then + close and delete the file. If the attacker recreates a file with the same + name, the state should be cleared. + + * Many reference monitors had accuracy or security bugs as a result of not + properly following the instructions. + + +## Example : Sample Implementation + +```py +class LPFile(): + def __init__(self, filename, create): + # globals + mycontext['debug'] = False + self.LPfile = openfile(filename, create) + self.pending_data = None + self.pending_offset = None + + def readat(self, bytes, offset): + # Read from the file using the sandbox's readat... + return self.LPfile.readat(bytes, offset) + + def writeat(self, data, offset): + # bug ? + self.LPfile.writeat(self.pending_data, self.pending_offset) + self.pending_data = data + self.pending_offset = offset + + def undo(self): + # bug ? + self.pending_data = None + self.pending_offset = None + + def close(self): + # bug ? + self.LPfile.close() +``` + +### Code Analysis + +Let's analyze some of the bugs in this implementation. + +According to the specifications in [Part +One](https://github.com/SeattleTestbed/docs/blob/master/EducationalAssignments/UndoPartOne.md), +the `writeat` function is designed to work such that the provided data is not +immediately written to the file. Instead, it should be stored in a "pending" +state. Here, the `writeat` writes to the file immediately without any form of +delay. Another bug is that on closing the file, the function doesn't check or +commit any `pending_data` to the file. Thus, if there was any `pending_data` +from the last `writeat` operation, it will be lost and not written to the file. + +There's no error handling for scenarios like writing with a negative offset, +writing beyond the end of the file, etc. Such error checks are essential to +prevent data corruption and unexpected behavior. Furthermore, there's no +mechanism to update or manage the current length of the file. This can cause +problems, especially if you're trying to prevent writing beyond the end of the +file or if you're trying to append data correctly. Moreover, this example +doesn't use any locks so you can introduce race conditions and perform a variety +of attacks. + + +## What to turn in? + + * Never raise unexpected errors or produce any output. Your program must + produce no output when run normally. + * Turn in the reference monitor that you have fixed. The name of the reference +monitor should be in the form of `reference_monitor_[ netid ].r2py`. All letters +must be lowercase. diff --git a/EducationalAssignments/UndoPartTwo.md b/EducationalAssignments/UndoPartTwo.md new file mode 100644 index 000000000..d649ce14d --- /dev/null +++ b/EducationalAssignments/UndoPartTwo.md @@ -0,0 +1,200 @@ +# Security layer testing and penetration + +In this assignment you will learn how to attack a reference monitor. The +reference monitor you will be testing uses the security layer framework +(encasement library, etc.) for the Seattle testbed. It is possible to do this +assignment separately, but it is recommended that this assignment be completed +after [Part One](./UndoPartOne.md). Either way you should already have a working +security layer or access to one. Testing the security layer is done by running a +series of test cases that an adversary may use to circumvent your system. This +assignment is intended to prepare you for thinking about security paradigms in a +functional way. The ideas of information, security and privacy have been +embedded into the steps of this assignment. + + +## Overview + +In this assignment you are a tester. You have been sent a bunch of reference +monitors that need testing before they are deployed. Your job will be to ensure +an attacker cannot circumvent these security layers. In order to do this, you +will attempt to write, and append invalid data. If you are able to do so, then +the security layer is not secure. The future of the system depends on your +ability to test code thoroughly! + +Three design paradigms are at work in this assignment: accuracy, efficiency, and +security. + + * Accuracy: The security layer should only stop certain actions from being + blocked. All other actions should be allowed. + + * Efficiency: The security layer should use a minimum number of resources, so + performance is not compromised. + + * Security: The attacker should not be able to circumvent the security layer. + + +Within the context of this assignment these design paradigms translate to: + + * Accuracy: The defense monitor should precisely and consistently manage file +operations. Only specific operations, such as `writeat`, should have their +behavior modified. All situations that are not described in the specifications +*must* match that of the underlying API. + + * Efficiency: The security layer should use a minimum number of resources, so +performance is not compromised. For example, you may not call `readat()` +everytime `writeat()` is called. It is permissable to call `readat()` upon +`fileopen()`, however. + + * Security: The defense layer should be robust against tampering and +circumvention. Attackers must not be able to bypass, disable, or exploit the +defense monitor's enhanced behaviors, ensuring the integrity and intended +functionality of file operations. + +You will submit a zip file containing all of the tests you have created. You +will gain points for every student's reference monitor you find a flaw in. It is +good if multiple tests of yours break a student's reference monitor, but you +gain the same number of tests whether one or more tests break the layer. + + +## Prerequisites + +This assignment assumes you have both the latest Python 2.7 and RepyV2 installed +on your computer. Please refer to the [SeattleTestbed Build +Instructions](../Contributing/BuildInstructions.md#prerequisites) for +information on how to get them. + + +### Helpful links + +The following links will aid students in becoming comfortable with Python, Repy +and seattle: + * Official [Python tutorial](http://docs.python.org/tutorial/) + * [Differences between RepyV2 and Python](../Programming/PythonVsRepyV2.md) + * List of [RepyV2 API calls](../Programming/RepyV2API.md) + + +## Testing security layers + +### Hypothesis, test case, counter example + +The goal of a good tester is to test hypotheses. A hypothesis is just a +scientific way of asking a question. The hypothesis of this assignment is "This +security layer is well designed." The questions you will ask when running your +test cases will always be the same + + * "Is this reference monitor secure?" + + * "Does this reference monitor hamper performance?" + + * "Does this reference monitor prevent actions that should be allowed?" + +Notice that these questions are parallels of the security paradigms: security, +efficiency and accuracy, respectively. + +If we can find a case where the hypothesis is false, then the security layer is +not secure. Such a case is referred to as a counter example. Hence all test +cases should be designed to test for these three types of flaws. + +#### Information on: Try, Except, Else, Finally + +The try, except, else and finally statements are part of **exception handling**. +For more information on exception handling please visit: + + * [http://docs.python.org/tutorial/errors.html] + * [http://wiki.python.org/moin/HandlingExceptions] + * [http://www.tutorialspoint.com/python/python_exceptions.htm] + +### Hints and Ideas for testing + +When writing your own tests it is important to test for a complete set of +possible penetrations. Keep in mind, it only takes one test case to break +through a security layer. Some of the things you may want to test for include: + + * threading + * multiple writes + +And more! Remember a good security layer can't be broken by anyone! Which is +all part of the fun! It's about solving a puzzle. First you make the puzzle - +write the security layer, then you solve the puzzle - try to bypass it. If your +puzzle is "good enough", no one will be able to break it, no matter what. + + +> Note: When creating tests, the best practice is to separate them into +> different files. Each test file should focus on testing one specific scenario +> or functionality. If there is a flaw in any part of a test file, we will +> discard the entire test file. This modular approach helps isolate issues - if +> there is a flaw in one test file, it will not affect or invalidate the other +> test files, since each file is independent and testing distinct functionality. + + +## Notes and Resources + + * The following link is an excellent source for information about security + layers: https://ssl.engineering.nyu.edu/papers/cappos_seattle_ccs_10.pdf + + * In repy `log` replaces `print` from python. Many students find this to be a + stumbling block. + + * Note that you should not assume that any files exist in your directory. You + should create any files (e.g., `testfile.txt`) yourself in your test program. + + * It's important to note that if a test has a flaw in any part of it, the + entire test will be considered invalid. So, it's advisable to break your + tests into different files. + +## How to run your tests on many reference monitors + +Create a directory that the security layers will write their files into. You +need to run repy with only access to this directory. You can write a test +program that does `log(str(listfiles()))` to see if you are in the right place. + +Have all the reference monitors to test and the test cases inside the same +directory where the `repy.py` file exists. + +* In the bash shell on Mac and Linux: +```bash +for referencemonitor in reference_monitor_*; do for testcase in _*; do python repy.py restrictions.default encasementlib.r2py $referencemonitor $testcase; done; done +``` + +* In the Command Prompt on Windows: +```cmd +FOR %r IN (reference_monitor_*) DO @FOR %a IN (_*) DO @python repy.py restrictions.default encasementlib.r2py %r %a +``` + +* In PowerShell on Windows: +```powershell +foreach ($referencemonitor in Get-ChildItem reference_monitor_*) { foreach ($testcase in Get-ChildItem _*) { python repy.py restrictions.default encasementlib.r2py $referencemonitor.Name $testcase.Name } } +``` + +This will print out the output from each program. Make sure that you replace +`` with your NetID. + +If you want to spot the reference monitor that failed during the test run, add +echo the name of each reference monitor before the inner loop, like so: + +* In the bash shell on Mac and Linux: +```bash +for referencemonitor in reference_monitor_*; do echo $referencemonitor under test; for testcase in _*; do python repy.py restrictions.default encasementlib.r2py $referencemonitor $testcase; done; done +``` + +* In the Command Prompt on Windows: +```cmd +FOR %r IN (reference_monitor_*) DO @(ECHO %r under test & FOR %a IN (_*) DO @python repy.py restrictions.default encasementlib.r2py %r %a) +``` + +* In PowerShell on Windows: +```powershell +foreach ($referencemonitor in Get-ChildItem reference_monitor_*) { Write-Host $referencemonitor.Name; foreach ($testcase in Get-ChildItem _*) { python repy.py restrictions.default encasementlib.r2py $referencemonitor.Name $testcase.Name } } +``` + +This will print out the name of each reference monitor before it starts +executing the test cases against it. + + +## What to turn in? + + * Never raise unexpected errors or produce any output. Your attack must produce + no output when run normally. + * Turn in the test cases used to attack a given reference monitor in a zip + file. The name of each testcase must match the following format: `[ net_id + ]_attackcase1.r2py`, `[ net_id ]_attackcase2.r2py`, etc. diff --git a/Programming/ProgrammersPage.md b/Programming/ProgrammersPage.md index 5c13f5bdf..603965baf 100644 --- a/Programming/ProgrammersPage.md +++ b/Programming/ProgrammersPage.md @@ -1,65 +1,66 @@ -# Programmer Portal +# Programmer Portal This page features information and references regarding how to use write programs on Seattle. Seattle nodes run code written in a special language called Repy. Repy is a subset of the [Python language](http://www.python.org/) (version 2.5 / version 2.6) invented specifically for Seattle. -## Before You Begin +## Before You Begin Make sure you have all the necessary tools installed: * **Step 1**: Python2.5 or Python2.6 must be installed in order to use Repy. [Download Python2.6](http://www.python.org/download/releases/2.6.4/) **WINDOWS USERS:** For instructions on how to check if you already have the correct version of Python installed, and for directions on how to install Python2.6, [wiki:InstallPythonOnWindows click here] * **Step 2**: You will have to download and install repy before starting the tutorials below: **[Download repy!](https://seattleclearinghouse.poly.edu/download/flibble/)** -## Tutorials +## Tutorials You can start with several different tutorials depending on your background. - * [wiki:PythonTutorial All of the Python You Need to Understand Repy (and None You Don't)]. Start here if you're new to Python and Repy! + * [PythonTutorial All of the Python You Need to Understand Repy (and None You Don't)](PythonTutorial.md) Start here if you're new to Python and Repy! - * [wiki:PythonVsRepy All of the Python You Need to Forget to Use Repy]. Use this tutorial if you already know Python. + * [PythonVsRepy All of the Python You Need to Forget to Use Repy](PythonVsRepy.md) Use this tutorial if you already know Python. + * [All of the Python You Need to Forget to Use RepyV2](PythonVsRepyV2.md) After doing one of the above tutorials, do the following tutorial to learn how to use Repy specific features and functionality: - * [wiki:RepyTutorial Repy Tutorial]. Try this tutorial to learn more about Repy features and the language API. + * [RepyTutorial Repy Tutorial](RepyTutorial.md) Try this tutorial to learn more about Repy features and the language API. + * [RePyV2 Tutorial](RepyV2Tutorial.md) If you would prefer to use Repy V2, it is worthwhile to read through the following tutorials: - * [wiki:RepyV2CheckoutAndUnitTests Checking out Repy V2 and Running the included unit tests] + * [RepyV2CheckoutAndUnitTests Checking out Repy V2 and Running the included unit tests](../Contributing/BuildInstructions.md) + * [RepyV2SecurityLayers How to use Security Layers with Repy V2](RepyV2SecurityLayers.md) - * [wiki:RepyV2SecurityLayers How to use Security Layers with Repy V2] +## Other Resources + * An easy way to start learning Seattle is to watch our [UnderstandingSeattle/DemoVideo five-minute demo](https://seattle.poly.edu/static/demo.mov). -## Other Resources - * An easy way to start learning Seattle is to watch our [wiki:UnderstandingSeattle/DemoVideo five-minute demo]. + * Use the [RepyApi Repy API Reference](RepyApi.md) as a quick guide to Repy's API once you are comfortable with the language. - * Use the [wiki:RepyApi Repy API Reference] as a quick guide to Repy's API once you are comfortable with the language. - - * Programmers writing code for Repy V2 may want to read about the [wiki:RepyV1vsRepyV2 differences between Repy V1 and Repy V2]. + * Programmers writing code for Repy V2 may want to read about the [RepyV1vsRepyV2 differences between Repy V1 and Repy V2](RepyV1vsRepyV2.md). * There is a growing list of library code you can download and use with Repy (see [seattlelib](http://seattle.poly.edu/svn/seattle/trunk/seattlelib/) in the Seattle repository for examples). Read more about RepyHelper for the details of how to include Repy code in Python programs. - * Documentation about the Seattle Standard Library can be seen at [wiki:SeattleLib] + * Documentation about the Seattle Standard Library can be seen at [SeattleLib](SeattleLib_v1/ProgrammerResources.md) - * Continuous Integration Guide for Seattle Projects: [wiki:ContinuousIntegrationGuide] + * Continuous Integration Guide for Seattle Projects: [ContinuousIntegrationGuide](../Contributing/ContinuousIntegration.md) -## Editing Repy files -### Vim +## Editing Repy files +### Vim João Moreno has also provided a [VIM syntax file](http://www.vim.org/scripts/script.php?script_id=2546) for Repy that will syntax color Repy programs. You can also choose to automatically color Repy code with python-mode in emacs. In your .emacs file, add the following line: ``` -(add-to-list 'auto-mode-alist '(" - +(add-to-list 'auto-mode-alist '(" + .repy$" . python-mode)) ``` -### gedit +### gedit In order to have proper syntax highlighting for Repy, `repy.lang` needs to be moved into the folder that stores the defined languages. It is a slightly modified version of the python language spec. * For system-wide changes, move it to: `/usr/share/gtksourceview-3.0/language-specs/repy.lang` * For a single user, move it to: `~/.local/share/gtksourceview-3.0/language-specs/repy.lang` -You can download `repy.lang` [raw-attachment:repy.lang here] \ No newline at end of file +You can download `repy.lang` [raw-attachment:repy.lang here] diff --git a/Programming/PythonVsRepyV2.md b/Programming/PythonVsRepyV2.md index 3e980b462..f07c0210a 100644 --- a/Programming/PythonVsRepyV2.md +++ b/Programming/PythonVsRepyV2.md @@ -92,7 +92,7 @@ super, unichr, unicode, vars, yield, __import__ ---- #### sys.argv -Programs written in Python use `sys.argv``` to access arguments to +Programs written in Python use `sys.argv` to access arguments to the file. In RepyV2 the variable `callargs` behaves the same as `sys.argv[1:]` @@ -112,6 +112,11 @@ if callfunc == "initialize": main() ``` +#### with +Programs written in Python can use the `with` statement in combination +with `__enter__` and `__exit__` methods. However, these constructs are +not available in RepyV2. + ## Python Modules diff --git a/Programming/SeattleLib_v1/AdvertiseObjects.repy.md b/Programming/SeattleLib_v1/AdvertiseObjects.repy.md index 688360278..b332f504a 100644 --- a/Programming/SeattleLib_v1/AdvertiseObjects.repy.md +++ b/Programming/SeattleLib_v1/AdvertiseObjects.repy.md @@ -1,19 +1,19 @@ -# AdvertiseObjects.repy +# AdvertiseObjects.repy Provides two objects which makes advertising more efficient. Can be used as an alternative to [wiki:Advertise.repy]. -LookupCache caches lookup results to increase performance in programs which calls lookup services frequently. +LookupCache caches lookup results to increase performance in programs which calls lookup services frequently. AdvertisePipe stores a list of (key, value) tuples and advertises them concurrently in a single thread. -# Functions +# Functions LookupCache ``` lookup(self, key, maxvals=100, lookuptype=['central','opendht','DOR'], concurrentevents=2, graceperiod=10, timeout=60) ``` - Exactly the same as [wiki:Advertise.repy]. - Note: + Exactly the same as [advertise.repy](advertise.repy.md). + Note: * self refers to its own cache. @@ -22,7 +22,7 @@ AdvertisePipe add(self, key, value) ``` Adds a (key, value) pair to the advertise pipe. - Note: + Note: * self refers to its own cache. * Each value added is given an unique handle, which is used in remove. @@ -31,14 +31,14 @@ add(self, key, value) remove(self, handle) ``` Removes a (key, value) pair from the advertise pipe. - Note: + Note: * handle is an unique identifier for each value in the advertise pipe. * self refers to its own cache -# Usage +# Usage no examples? -# Includes -[wiki:Advertise.repy] +# Includes +[advertise.repy](advertise.repy.md) diff --git a/Programming/SeattleLib_v1/ConcurrencyAndParallelism.md b/Programming/SeattleLib_v1/ConcurrencyAndParallelism.md index 6fe332a51..36e346070 100644 --- a/Programming/SeattleLib_v1/ConcurrencyAndParallelism.md +++ b/Programming/SeattleLib_v1/ConcurrencyAndParallelism.md @@ -1,7 +1,7 @@ -### Description +### Description The parallelism services offered by Seattle are purely for the client's needs and not required in any way. Running parallel processes are more efficient and may be of interest to some users, but adds a certain amount of overhead since locking must be implemented to prevent crashes and errors in critical sections of code. For this reason, libraries like [wiki:SeattleLib/semaphore.repy] and [wiki:SeattleLib/cv.repy] to help the client implement locks. Many modules here also have Python equivalents. These are linked appropriately. -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/Cryptography.md b/Programming/SeattleLib_v1/Cryptography.md index 17e1bd9b1..242382663 100644 --- a/Programming/SeattleLib_v1/Cryptography.md +++ b/Programming/SeattleLib_v1/Cryptography.md @@ -1,4 +1,4 @@ -### Description +### Description Seattle supports various cryptography choices. Since Seattle includes a lot of network communication, encrypting data is very important. This prevents 3rd party interference. @@ -6,4 +6,4 @@ Many of the modules in this category are ported directly from Python equivalents Note also a few modules are very experimental in nature and may not have full functionality. All of these discrepancies will be noted as well per section. -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/DORadvertise.repy.md b/Programming/SeattleLib_v1/DORadvertise.repy.md index 7ea46f52e..7e5bd86fe 100644 --- a/Programming/SeattleLib_v1/DORadvertise.repy.md +++ b/Programming/SeattleLib_v1/DORadvertise.repy.md @@ -1,11 +1,11 @@ -# DORadvertise.repy +# DORadvertise.repy Another method of advertising nodes in the Seattle library. This module advertises to the Digital Object Registry run by CNRI. -### Functions +### Functions ``` DORadvertise_announce(key, value, ttlval, timeout=None) ``` Adds a (key, value) pair into the DOR. - Notes: + Notes: * ttlval describes the length of time in seconds the tuple exists in the DOR. * Exceptions are raised if there are errors within the XML-RPC client. @@ -15,18 +15,20 @@ DORadvertise_announce(key, value, ttlval, timeout=None) DORadvertise_lookup(key, maxvals=100, timeout=None) ``` Looks up a stored value under the key in the DOR. - Notes: + Notes: * maxvals is the maximum number of values returned. * timeout is the number of seconds spent before the process quits. -### Usage +### Usage ???couldn't find any? -### Include -[wiki:SeattleLib/sockettimeout.repy] +### Include +[SeattleLib/sockettimeout.repy](sockettimeout.repy.md) -[wiki:SeattleLib/httpretrieve.repy] +[SeattleLib/httpretrieve.repy](httpretrieve.repy.md) -[wiki:SeattleLib/xmlparse.repy] +[SeattleLib/xmlparse.repy](xmlparse.repy.md) + +[Back to NodeAdvertising](NodeAdvertising.md) diff --git a/Programming/SeattleLib_v1/DataEncoding.md b/Programming/SeattleLib_v1/DataEncoding.md index 7d2e9c48d..b44a31014 100644 --- a/Programming/SeattleLib_v1/DataEncoding.md +++ b/Programming/SeattleLib_v1/DataEncoding.md @@ -1,5 +1,5 @@ -### Description +### Description These services are used to protect transmission of data within the network. Encryption is also necessary in order to allow for serialization of the data, which is essential for data transmission as well. Programmers should take care to encrypt all their packets. -The Seattle Standard Library supports two different types of encoding methods: binary to ascii ([wiki:SeattleLibbinascii.repy]) and base64 ([wiki:SeattleLibbase64.repy]). The serialization module ([wiki:SeattleLib/serialize.repy]) allows for serialization and deserialization of various data types. \ No newline at end of file +The Seattle Standard Library supports two different types of encoding methods: binary to ascii ([binascii.repy](binascii.repy.md)) and base64 ([base64.repy](base64.repy.md)). The serialization module ([serialize.repy](serialize.repy.md)) allows for serialization and deserialization of various data types. diff --git a/Programming/SeattleLib_v1/DataRetrieval.md b/Programming/SeattleLib_v1/DataRetrieval.md index d0d0a411b..008264801 100644 --- a/Programming/SeattleLib_v1/DataRetrieval.md +++ b/Programming/SeattleLib_v1/DataRetrieval.md @@ -1,7 +1,7 @@ -### Description +### Description -All the modules in category deal with data retrieval of some form. Since Seattle is a P2P based client, its often necessary to know about a specific node: where it is, what is its IP address, etc. In addition to [wiki:SeattleLib/domainnameinfo.repy], which simply returns the country of origin, Seattle also provides the [wiki:SeattleLib/geoip_client.repy], which returns specific location information about the client, etc. +All the modules in category deal with data retrieval of some form. Since Seattle is a P2P based client, its often necessary to know about a specific node: where it is, what is its IP address, etc. In addition to [domainnameinfo.repy](domainnameinfo.repy.md), which simply returns the country of origin, Seattle also provides the [geoip_client.repy](geoip_client.repy.md), which returns specific location information about the client, etc. -Seattle also provides a basic abstraction to the HTTP protocol, see [wiki:SeattleLib/httpserver.repy]. +Seattle also provides a basic abstraction to the HTTP protocol, see [httpserver.repy](httpserver.repy.md). -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/NAT_advertisement.repy.md b/Programming/SeattleLib_v1/NAT_advertisement.repy.md index 9c611b4f8..e6c424d39 100644 --- a/Programming/SeattleLib_v1/NAT_advertisement.repy.md +++ b/Programming/SeattleLib_v1/NAT_advertisement.repy.md @@ -1,8 +1,8 @@ -# NAT_advertisement.repy +# NAT_advertisement.repy -Abstracts the task of looking up and advertising servers and forwarders. Allows for those using NAT_layer to advertise. Utilizes [wiki:Advertise.repy] to achieve this. +Abstracts the task of looking up and advertising servers and forwarders. Allows for those using NAT_layer to advertise. Utilizes [advertise.repy](advertise.repy.md) to achieve this. -### Functions +### Functions ``` nat_forwarder_advertise(ip, serverport, clientport) @@ -33,12 +33,12 @@ nat_server_list_lookup(key) nat_toggle_advertisement(enabled, threadRun=True) ``` Toggles the state of the advertisement. - Notes: + Notes: * threadRun controls the advertisement thread. -### Usage +### Usage ??? -### Includes -[wiki:Advertise.repy] \ No newline at end of file +### Includes +[advertise.repy](advertise.repy.md) diff --git a/Programming/SeattleLib_v1/NetworkCommunication.md b/Programming/SeattleLib_v1/NetworkCommunication.md index 0b8000de1..40608913d 100644 --- a/Programming/SeattleLib_v1/NetworkCommunication.md +++ b/Programming/SeattleLib_v1/NetworkCommunication.md @@ -1,7 +1,7 @@ -### Description +### Description -This section describes the all important tools necessary for dealing with sockets. For instance, we can use the modules in the section to find out critical information about VMs, which are elements that computers whose resources you can access ([wiki:SeattleLib/getvesselsresources.repy]). We also have basic control over socket connections such as forcing hanging connections to quit ([wiki:SeattleLib/sockettimeout.repy]). +This section describes the all important tools necessary for dealing with sockets. For instance, we can use the modules in the section to find out critical information about VMs, which are elements that computers whose resources you can access ([getvesselsresources.repy](getvesselsresources.repy.md)). We also have basic control over socket connections such as forcing hanging connections to quit ([sockettimeout.repy](sockettimeout.repy.md)). -[wiki:SeattleLib/nmclient.repy] is a also included here. It contains the backend to what could be an very useful external node manager, but by itself is not fully functional. +[nmclient.repy](nmclient.repy.md) is a also included here. It contains the backend to what could be an very useful external node manager, but by itself is not fully functional. -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/NodeAdvertising.md b/Programming/SeattleLib_v1/NodeAdvertising.md index 098f434e1..b1a655860 100644 --- a/Programming/SeattleLib_v1/NodeAdvertising.md +++ b/Programming/SeattleLib_v1/NodeAdvertising.md @@ -1,13 +1,13 @@ -### Description +### Description -Seattle uses a node based service where available resources are stored by value and key pairs. All available resources (otherwise known as nodes) are thus hashed to a global store (otherwise known as advertising). The Seattle Standard Library provides three different methods of advertising nodes: [wiki:SeattleLibcentralizedadvertise.repy], [wiki:SeattleLib/openDHTadvertise.repy], or [wiki:SeattleLib/DORadvertise.repy]. +Seattle uses a node based service where available resources are stored by value and key pairs. All available resources (otherwise known as nodes) are thus hashed to a global store (otherwise known as advertising). The Seattle Standard Library provides three different methods of advertising nodes: [SeattleLibcentralizedadvertise.repy](centralizedadvertise.repy.md), [openDHTadvertise.repy](openDHTadvertise.repy.md), or [DORadvertise.repy](DORadvertise.repy.md). -[wiki:SeattleLibcentralizedadvertise.repy] uses a centralized hash table to store all the values, which runs on the main Seattle server. This may be desirable to users who do not want to depend on the OpenDHT client in case of failure, etc. +[SeattleLibcentralizedadvertise.repy](centralizedadvertise.repy.md) uses a centralized hash table to store all the values, which runs on the main Seattle server. This may be desirable to users who do not want to depend on the OpenDHT client in case of failure, etc. -[wiki:SeattleLib/openDHTadvertise.repy] uses the OpenDHT client to store key value pairs. +[openDHTadvertise.repy](openDHTadvertise.repy.md) uses the OpenDHT client to store key value pairs. -[wiki:SeattleLib/DORadvertise.repy] uses the CNRI service. +[DORadvertise.repy](DORadvertise.repy.md). uses the CNRI service. -One of these services may be chosen for exclusive use, but [wiki:SeattleLibadvertise.repy] is the most common choice, as it combines all three services and allows the user to pick a specific implementation of node advertising. +One of these services may be chosen for exclusive use, but [advertise.repy](advertise.repy.md) is the most common choice, as it combines all three services and allows the user to pick a specific implementation of node advertising. -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/ProgrammerResources.md b/Programming/SeattleLib_v1/ProgrammerResources.md index d92c4af85..01ad31f5f 100644 --- a/Programming/SeattleLib_v1/ProgrammerResources.md +++ b/Programming/SeattleLib_v1/ProgrammerResources.md @@ -1,13 +1,13 @@ -### Description +### Description -This section contains all the extraneous data structures that Seattle supports, in addition to all the various backend modules that makes seash work. There are several utilities which can help with basic programming tasks such as [wiki:SeattleLib/math.repy], [wiki:SeattleLib/random.repy], and etc. Notably modules include: +This section contains all the extraneous data structures that Seattle supports, in addition to all the various backend modules that makes seash work. There are several utilities which can help with basic programming tasks such as [math.repy](math.repy.md), [random.repy](random.repy.md), and etc. Notably modules include: -[wiki:SeattleLib/safe_eval.repy] allows one to safely evaluate strings, free from the context of whatever it is in. +[safe_eval.repy](safe_eval.repy.md) allows one to safely evaluate strings, free from the context of whatever it is in. -[wiki:SeattleLibargparse.repy] checks command line arguments and separates in a way that is usable. This utility can be used within the context of whatever program it's in, making this a very useful module. +[argparse.repy](argparse.repy.md) checks command line arguments and separates in a way that is usable. This utility can be used within the context of whatever program it's in, making this a very useful module. -[wiki:SeattleLib/urlparse.repy] parses urls for network communication purposes. This is primarily used in the XML parsing section. +[urlparse.repy](urlparse.repy.md) parses urls for network communication purposes. This is primarily used in the XML parsing section. Please note that the The backend modules that are located here are mostly ones that the average user would not have to consider. -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/SeattleLib.md b/Programming/SeattleLib_v1/SeattleLib.md index 6d41afa8b..1ff825b53 100644 --- a/Programming/SeattleLib_v1/SeattleLib.md +++ b/Programming/SeattleLib_v1/SeattleLib.md @@ -1,80 +1,80 @@ -# Seattle Standard Library (SeattleLib) +# Seattle Standard Library (SeattleLib) This pages contains documentation about the Seattle standard library (called "SeattleLib"). The libraries are broken up by similar category for convenience. Click on each category for overarching descriptions and more information. The bolded links are modules are meant for direct usage by the client. They often aggregate multiple services. - * [wiki:SeattleLib/NodeAdvertising Node Advertising] (click for a description) - * **[wiki:SeattleLib/advertise.repy advertise.repy]** - * [wiki:SeattleLib/AdvertiseObjects.repy AdvertiseObjects.repy] - * [wiki:SeattleLib/centralizedadvertise.repy centralizedadvertise.repy] - * [wiki:SeattleLib/DORadvertise.repy DORadvertise.repy] - * [wiki:SeattleLib/NAT_advertisement.repy NAT_advertisement.repy] - * [wiki:SeattleLib/openDHTadvertise.repy openDHTadvertise.repy] + ## [Node Advertising](NodeAdvertising.md) (click for a description) + * **[advertise.repy](advertise.repy.md)** + * [AdvertiseObjects.repy](AdvertiseObjects.repy.md) + * [centralizedadvertise.repy](centralizedadvertise.repy.md) + * [DORadvertise.repy](DORadvertise.repy.md) + * [NAT_advertisement.repy](NAT_advertisement.repy.md) + * [openDHTadvertise.repy](openDHTadvertise.repy.md) - * [wiki:SeattleLib/Cryptography Cryptography] (click for a description) - * [wiki:SeattleLib/md5py.repy md5py.repy] - * [wiki:SeattleLib/pycryptorsa.repy pycryptorsa.repy] - * [wiki:SeattleLib/pyDes.repy pyDes.repy] - * [wiki:SeattleLib/rsa.repy rsa.repy] - * [wiki:SeattleLib/sha.repy sha.repy] - * **[wiki:SeattleLib/sshkey.repy sshkey.repy]** - * **[wiki:SeattleLib/sshkey_paramiko.repy sshkey_paramiko.repy]** - * [wiki:SeattleLib/signeddata.repy signeddata.repy] + ## [Cryptography](Cryptography.md) (click for a description) + * [md5py.repy](md5py.repy.md) + * [pycryptorsa.repy](pycryptorsa.repy.md) + * [wpyDes.repy](pyDes.repy.md) + * [rsa.repy](rsa.repy.md) + * [sha.repy](sha.repy.md) + * **[sshkey.repy](sshkey.repy.md)** + * **[sshkey_paramiko.repy](sshkey_paramiko.repy.md)** + * [signeddata.repy](signeddata.repy.md) - * [wiki:SeattleLib/Time Time] (click for a description) - * [wiki:SeattleLib/ntp_time.repy ntp_time.repy] - * [wiki:SeattleLib/tcp_time.repy tcp_time.repy] - * **[wiki:SeattleLib/time.repy time.repy]** - * [wiki:SeattleLib/time_interface.repy time_interface.repy] + ## [Time](Time.md) (click for a description) + * [ntp_time.repy](ntp_time.repy.md) + * [tcp_time.repy](tcp_time.repy.md) + * **[time.repy](time.repy.md)** + * [time_interface.repy](time_interface.repy.md) - * [wiki:SeattleLib/DataEncoding Data encoding] (click for a description) - * [wiki:SeattleLib/base64.repy base64.repy] - * [wiki:SeattleLib/binascii.repy binascii.repy] - * **[wiki:SeattleLib/deserialize.repy deserialize.repy]** - * **[wiki:SeattleLib/serialize.repy serialize.repy]** + ## [Data encoding](DataEncoding.md) (click for a description) + * [base64.repy](base64.repy.md) + * [binascii.repy](binascii.repy.md) + * **[wiki::deserialize.repy]** + * **[wserialize.repy](serialize.repy.md)** - * [wiki:SeattleLib/UrlParsingAndXml URL parsing / XML] (click for a description) - * [wiki:SeattleLib/httpretrieve.repy httpretrieve.repy] - * [wiki:SeattleLib/xmlparse.repy xmlparse.repy] - * [wiki:SeattleLib/xmlrpc_client.repy xmlrpc_client.repy] - * [wiki:SeattleLib/xmlrpc_common.repy xmlrpc_common.repy] - * **[wiki:SeattleLib/xmlrpc_server.repy xmlrpc_server.repy]** - * [wiki:SeattleLib/urllib.repy urllib.repy] + ## [URL parsing / XML](UrlParsingAndXml.md) (click for a description) + * [httpretrieve.repy](httpretrieve.repy.md) + * [xmlparse.repy](xmlparse.repy.md) + * [xmlrpc_client.repy](xmlrpc_client.repy.md) + * [xmlrpc_common.repy](xmlrpc_common.repy.md) + * **[xmlrpc_server.repy](xmlrpc_server.repy.md)** + * [urllib.repy](urllib.repy.md) - * [wiki:SeattleLib/ConcurrencyAndParallelism Concurrency / Parallelism] (click for a description) - * [wiki:SeattleLib/cv.repy cv.repy] - * [wiki:SeattleLib/parallelize.repy parallelize.repy] - * [wiki:SeattleLib/semaphore.repy semaphore.repy] - * **[wiki:SeattleLib/uniqueid.repy uniqueid.repy]** + ## [Concurrency / Parallelism](ConcurrencyAndParallelism.md) (click for a description) + * [cv.repy](cv.repy.md) + * [parallelize.repy](parallelize.repy.md) + * [semaphore.repy](semaphore.repy.md) + * **[uniqueid.repy](uniqueid.repy.md)** - * [wiki:SeattleLib/DataRetrieval Data retrieval] (click for a description) - * [wiki:SeattleLib/domainnameinfo.repy domainnameinfo.repy] - * [wiki:SeattleLib/geoip_client.repy geoip_client.repy] - * [wiki:SeattleLib/httpserver.repy httpserver.repy] - * [wiki:SeattleLib/servicelookup.repy servicelookup.repy] + ## [Data retrieval](DataRetrieval.md) (click for a description) + * [domainnameinfo.repy](domainnameinfo.repy.md) + * [geoip_client.repy](geoip_client.repy.md) + * [httpserver.repy](httpserver.repy.md) + * [servicelookup.repy](servicelookup.repy.md) - * [wiki:SeattleLib/NetworkCommunication Network communication] (click for a description) - * [wiki:SeattleLib/getvesselsresources.repy getvesselsresources.repy] + ## [Network communication](NetworkCommunication.md) (click for a description) + * [getvesselsresources.repy](getvesselsresources.repy.md) * [wiki:SeattleLib/Multiplexer.repy Multiplexer.repy] - * [wiki:SeattleLib/nmclient.repy nmclient.repy] - * **[wiki:SeattleLib/NATLayer_rpc.repy NATLayer_rpc.repy]** - * **[wiki:SeattleLib/sockettimeout.repy sockettimeout.repy]** - * **[wiki:SeattleLib/session.repy session.repy]** + * [nmclient.repy](nmclient.repy.md) + * **[NATLayer_rpc.repy](NATLayer_rpc.repy.md)** + * **[sockettimeout.repy](sockettimeout.repy.md)** + * **[session.repy](session.repy.md)** - * [wiki:SeattleLib/ProgrammerResources Programmer resources] (click for a description) - * [wiki:SeattleLib/argparse.repy argparse.repy] - * [wiki:SeattleLib/dylink.repy dylink.repy] - * [wiki:SeattleLib/listops.repy listops.repy] - * [wiki:SeattleLib/math.repy math.repy] - * [wiki:SeattleLib/priority_queue.repy priority_queue.repy] - * **[wiki:SeattleLib/repyunit.repy repyunit.repy]** - * [wiki:SeattleLib/random.repy random.repy] - * [wiki:SeattleLib/repypp.py repypp.py] - * [wiki:SeattleLib/safe_eval.repy safe_eval.repy] - * [wiki:SeattleLib/strace.py strace.py] - * [wiki:SeattleLib/textops.py textops.py] - * [wiki:SeattleLib/urlparse.repy urlparse.repy] - * [wiki:SeattleLib/dnscommon.repy dnscommon.repy] - * [wiki:SeattleLib/bundle.repy bundle.repy] \ No newline at end of file + ## [Programmer resources](ProgrammerResources.md) (click for a description) + * [argparse.repy](argparse.repy.md) + * [dylink.repy](dylink.repy.md) + * [listops.repy](listops.repy.md) + * [math.repy](math.repy.md) + * [priority_queue.repy](priority_queue.repy.md) + * **[repyunit.repy](repyunit.repy.md)** + * [random.repy](random.repy.md) + * [repypp.py](repypp.py.md) + * [safe_eval.repy](safe_eval.repy.md) + * [strace.py](strace.py.md) + * [textops.py](textops.py.md) + * [urlparse.repy](urlparse.repy.md) + * [dnscommon.repy](dnscommon.repy.md) + * [bundle.repy](bundle.repy.md) diff --git a/Programming/SeattleLib_v1/Time.md b/Programming/SeattleLib_v1/Time.md index a86e00010..3572cca3d 100644 --- a/Programming/SeattleLib_v1/Time.md +++ b/Programming/SeattleLib_v1/Time.md @@ -1,7 +1,9 @@ -### Description +### Description Like many services, Seattle contains its own time module, which provides various time related functions like getting the time and updating the time. Seattle provides both TCP and NTP time services. TCP, otherwise known as Transmission Control Protocol, utilizes timestamps to keep track of time. NTP, or Network Time Protocol, synchronizes clocks on different machines by using various jitter buffers. -In any case, all programmers should include [wiki:SeattleLib/time.repy], which ties both services into one. \ No newline at end of file +In any case, all programmers should include [time.repy](time.repy.md), which ties both services into one. + +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/UrlParsingAndXml.md b/Programming/SeattleLib_v1/UrlParsingAndXml.md index a842d3f3f..95356d9fb 100644 --- a/Programming/SeattleLib_v1/UrlParsingAndXml.md +++ b/Programming/SeattleLib_v1/UrlParsingAndXml.md @@ -1,9 +1,9 @@ -### Description +### Description This class of modules supported by the Seattle Standard Library deal with URLs and XML. In most cases, Seattle utilizes the XML-RPC protocol to communicate between computers. The remote procedure calls (RPC) are achieved by using HTTP requests. In XML-RPC, the parameter for the the HTTP requests can be nested, in this case with XML. -In any case that the XML-RPC service is desirable, [wiki:SeattleLib/xmlrpc_server.repy] should be used. There rest of the modules described here which are mostly helper modules to [wiki:SeattleLib/xmlrpc_server.repy]. +In any case that the XML-RPC service is desirable, [xmlrpc_server.repy](xmlrpc_server.repy.md) should be used. There rest of the modules described here which are mostly helper modules to [xmlrpc_server.repy](xmlrpc_server.repy.md). Note also there are many parallels between these modules and equivalent Python modules. Please email [shurui@cs.washington.edu] if I have failed to link any. -[Back to SeattleLibWiki] \ No newline at end of file +[Back to SeattleLibWiki](../) diff --git a/Programming/SeattleLib_v1/advertise.repy.md b/Programming/SeattleLib_v1/advertise.repy.md index 4c81b9139..2b57875b6 100644 --- a/Programming/SeattleLib_v1/advertise.repy.md +++ b/Programming/SeattleLib_v1/advertise.repy.md @@ -1,8 +1,8 @@ -# advertise.repy +# advertise.repy This module allows for different node to be announced via different services (central advertise service, OpenDHT, or both). -### Functions +### Functions ``` advertise_announce(key, value, ttlval, concurrentevents=2, graceperiod=10, timeout=60) ``` @@ -13,7 +13,7 @@ advertise_announce(key, value, ttlval, concurrentevents=2, graceperiod=10, timeo * concurrentevents is how many services to announce in parallel. * graceperiod and timeout are both optional parameters. -### Example Usage +### Example Usage ``` advertise_lookup(key, maxvals=100, lookuptype=None, concurrentevents=2, graceperiod=10, timeout=60) @@ -24,7 +24,7 @@ advertise_lookup(key, maxvals=100, lookuptype=None, concurrentevents=2, graceper * lookuptype defaults to look in all types. * lookuptype, concurrentevents, graceperiod, and timeout are all optional. -### Usage +### Usage ``` #retrieve node list from advertise_lookup @@ -36,15 +36,15 @@ node_list = advertise_lookup(node_state_pubkey, maxvals = 10*1024*1024, lookupty advertise_announce(advertisekey, str(my_name), adTTL) ``` -### Includes -[wiki:SeattleLib/listops.repy] +### Includes +[SeattleLib/listops.repy](listops.repy.md) -[wiki:SeattleLib/openDHTadvertise.repy] +[SeattleLib/openDHTadvertise.repy](openDHTadvertise.repy.md) -[wiki:SeattleLibcentralizedadvertise.repy] +[SeattleLibcentralizedadvertise.repy](centralizedadvertise.repy.md) -[wiki:SeattleLib/DORadvertise.repy] - -[wiki:SeattleLib/parallelize.repy] +[SeattleLib/DORadvertise.repy](DORadvertise.repy.md) +[SeattleLib/parallelize.repy](parallelize.repy.md) +[Back to NodeAdvertising](NodeAdvertising.md) diff --git a/Programming/SeattleLib_v1/centralizedadvertise.repy.md b/Programming/SeattleLib_v1/centralizedadvertise.repy.md index 8f5cb1338..cc2bc8939 100644 --- a/Programming/SeattleLib_v1/centralizedadvertise.repy.md +++ b/Programming/SeattleLib_v1/centralizedadvertise.repy.md @@ -1,8 +1,8 @@ -# centralizedadvertise.repy +# centralizedadvertise.repy This module provides a hash table service for nodes. Adds and removes entries to a centralized hash table. This service runs on seattle.cs, which is also known as satya.cs. See CentralizedAdvertiseService for more details. -### Functions +### Functions @@ -24,7 +24,7 @@ centralizedadvertise_lookup(key, maxvals=100) * maxvals must be a positive integer that describes how many values to return. * Network / Timeout exception are raised if there are connection errors. -### Example Usage +### Example Usage ``` #advertize that the current client has started @@ -32,9 +32,10 @@ my_advetisement_info = getmyip() + ":"+ str(mycontext['myport']) + ":" + str(key centralizedadvertise_announce(mycontext['experiment_name'], my_advetisement_info, ADVERTISE_PERSIST) ``` -### Includes -[wiki:SeattleLib/sockettimeout.repy] -[wiki:SeattleLib/serialize.repy] +### Includes +[sockettimeout.repy](sockettimeout.repy.md) + +[serialize.repy](serialize.repy.md) diff --git a/Programming/SeattleLib_v1/httpserver.repy.md b/Programming/SeattleLib_v1/httpserver.repy.md index 4c2961aaf..fbd98b134 100644 --- a/Programming/SeattleLib_v1/httpserver.repy.md +++ b/Programming/SeattleLib_v1/httpserver.repy.md @@ -1,18 +1,18 @@ -# httpserver.repy +# httpserver.repy This library abstracts away the details of the HTTP protocol, providing an alternative to calling a user-supplied function on each request. The return value of the user-supplied function determines the response that is sent to the HTTP client. This allows the end user to not have to worry about the semantics of communicating with an HTTP server. In many ways, this module is analogous to the http.server module in Python documentation. See http://docs.python.org/py3k/library/http.server.html. -### Classes & Functions +### Classes & Functions ``` def httpserver_registercallback(addresstuple, cbfunc): ``` Registers a callback function on the (host, port). - Notes: + Notes: * addresstuple is an address 2-tuple to bind to: ('host', port). * cbfunc is the callback function to process requests. It takes one argument, which is a dictionary describing the HTTP request. It looks like this (just an example): @@ -49,21 +49,21 @@ def httpserver_stopcallback(callbackid): ``` Removes an existing callback function, i.e. stopping the server. - Notes: + Notes: * callbackid is the id returned by httpserver_registercallback(). * Raises IndexError or KeyError if the id is invalid or has already been deleted. -### Includes +### Includes -[wiki:SeattleLib/urllib.repy] +[wiki:SeattleLib/urllib.repy](urllib.repy.md) -[wiki:SeattleLib/urlparse.repy] +[wiki:SeattleLib/urlparse.repy](urlparse.repy.md) -[wiki:SeattleLib/uniqueid.repy] +[wiki:SeattleLib/uniqueid.repy](uniqueid.repy.md) -[wiki:SeattleLib/sockettimeout.repy] +[wiki:SeattleLib/sockettimeout.repy](sockettimeout.repy.md) -[wiki:SeattleLib/httpretrieve.repy] +[wiki:SeattleLib/httpretrieve.repy](httpretrieve.repy.md) diff --git a/Programming/SeattleLib_v1/openDHTadvertise.repy.md b/Programming/SeattleLib_v1/openDHTadvertise.repy.md index a438ea19d..ab2ec7df1 100644 --- a/Programming/SeattleLib_v1/openDHTadvertise.repy.md +++ b/Programming/SeattleLib_v1/openDHTadvertise.repy.md @@ -1,7 +1,7 @@ -# openDHTadvertise.repy +# openDHTadvertise.repy Utilizes OpenDHT to advertise availability of nodes. -### Functions +### Functions ``` openDHTadvertise_announce(key, value, ttlval, concurrentevents=5, proxiestocheck=5, timeout=None) ``` @@ -29,14 +29,14 @@ openDHTadvertise_get_proxy_list(maxnumberofattempts=5, concurrentevents=5) ``` Retrieves a list of active OpenDHT proxies. -### Example Usage +### Example Usage ...? only located in .svn directories? -### Includes -[wiki:SeattleLib/random.repy] +### Includes +[include random.repy](random.repy.md) -[wiki:include sha.repy] +[include sha.repy](sha.repy.md) -[wiki:include xmlrpc_client.repy] +[include xmlrpc_client.repy](xmlrpc_client.repy.md) -[wiki:include parallelize.repy] +[include parallelize.repy](parallelize.repy.md) diff --git a/Programming/SeattleLib_v1/rsa.repy.md b/Programming/SeattleLib_v1/rsa.repy.md index 0f7ca28d9..8486463c2 100644 --- a/Programming/SeattleLib_v1/rsa.repy.md +++ b/Programming/SeattleLib_v1/rsa.repy.md @@ -1,11 +1,11 @@ -# rsa.repy +# rsa.repy This is the main interface for using the ported RSA implementation. See http://en.wikipedia.org/wiki/RSA for more details. This port to repy is primarily taken from Dwayne C. Litzenberger's (www.dlitz.net) python code. -### Functions +### Functions ``` @@ -14,10 +14,10 @@ def rsa_gen_pubpriv_keys(bitsize): Will generate a new key object with a key size of the argument bitsize and return it. A recommended value would be 1024. - Notes: + Notes: * bitsize is the number of bits that the key should have. This means the modulus (publickey - n) will be in the range [2**(bitsize - 1), 2**(bitsize) - 1] - * Will return a key object that rsa_encrypt, rsa_decrypt, rsa_sign, and rsa_validate can use to preform their tasts. + * Will return a key object that rsa_encrypt, rsa_decrypt, rsa_sign, and rsa_validate can use to preform their tasts. @@ -29,7 +29,7 @@ def rsa_encrypt(message, publickey): Will use the key to encrypt the message string. If the string is to large to be encrypted it will be broken into chunks that are suitable for the keys modulus by the _rsa_chopstring function. - Notes: + Notes: * message is the string to be encrypted, there is no restriction on size. * publickey must be a valid publickey dictionary of the form {'n': 1.., 'e': 6..} with the keys being 'n' and 'e'. @@ -49,7 +49,7 @@ def rsa_decrypt(cypher, privatekey): If the plaintext string was to large to be encrypted it will use _rsa_gluechops and _rsa_unpicklechops to reassemble the origional plaintext after the individual peices are decrypted. - Notes: + Notes: * cypher is the encrypted string that was returned by rsa_encrypt. Example: " 142247030 31373650 44827705" * privatekey must be a valid privatekey dictionary of the form {'d':1.., 'p':1.. 'q': 1..} with the keys being 'd', 'p', and 'q'. @@ -67,7 +67,7 @@ def rsa_sign(message, privatekey): Will use the key to sign the plaintext string. - Notes: + Notes: * message is the string to be encrypted, there is no restriction on size. * publickey must be a valid publickey dictionary of the form {'n': 1.., 'e': 6..} with the keys being 'n' and 'e'. @@ -85,7 +85,7 @@ def rsa_verify(cypher, publickey): Will use the private key to decrypt the cypher string. - Notes: + Notes: * If the plaintext string was to large to be encrypted it will use _rsa_gluechops and _rsa_unpicklechops to reassemble the origional plaintext after the individual peices are decrypted. * cypher is the encrypted string that was returned by rsa_encrypt. Example: " 142247030 31373650 44827705" @@ -103,7 +103,7 @@ def rsa_is_valid_privatekey(key): This tries to determine if a key is valid. If it returns False, the key is definitely invalid. If True, the key is almost certainly valid. - Notes: + Notes: * key is a dictionary of the form {'d':1.., 'p':1.. 'q': 1..} with the keys 'd', 'p', and 'q' * If the key is valid, True will be returned. Otherwise False will be returned. @@ -125,7 +125,7 @@ def rsa_publickey_to_string(publickey): ``` To convert a publickey to a string. It will read the publickey which should a dictionary, and return it in the appropriate string format. - Notes: + Notes: * Must be a valid publickey dictionary of the form {'n': 1.., 'e': 6..} with the keys 'n' and 'e'. * Raises ValueError if the publickey is invalid. @@ -140,7 +140,7 @@ def rsa_string_to_publickey(mystr): To read a private key string and return a dictionary in the appropriate format: {'n': 1.., 'e': 6..} with the keys 'n' and 'e'. - Notes: + Notes: * mystr is a string containing the publickey, should be in the format created by the function rsa_publickey_to_string. Example if e=3 and n=21, mystr = "3 21" * Raises ValueError if the string containing the privateky is in a invalid format. @@ -173,7 +173,7 @@ def rsa_privatekey_to_file(key,filename): To write a privatekey to a file. It will convert the privatekey which should a dictionary, to the appropriate format and write it to a file, so that it can be read by rsa_file_to_privatekey. - Notes: + Notes: * privatekey must be a valid privatekey dictionary of the form {'d':1.., 'p':1.. 'q': 1..} with the keys 'd', 'p', and 'q' * filename is the string containing the name for the desired publickey file. @@ -189,7 +189,7 @@ def rsa_file_to_privatekey(filename): To read a file containing a key that was created with rsa_privatekey_to_file and return it in the appropriate format: {'d':1.., 'p':1.. 'q': 1..} with the keys 'd', 'p', and 'q'. - Notes: + Notes: * filename is the name of the file containing the privatekey. * Raises ValueError if the file contains an invalid private key string. @@ -217,8 +217,9 @@ def rsa_matching_keys(privatekey, publickey): ``` -### Includes +### Includes -[wiki:SeattleLib/random.repy] -[wiki:SeattleLib/pycryptorsa.repy] +[random.repy](random.repy.md) + +[pycryptorsa.repy](pycryptorsa.repy.md) diff --git a/Programming/SeattleLib_v1/semaphore.repy.md b/Programming/SeattleLib_v1/semaphore.repy.md index 6b90de90a..ad92f318e 100644 --- a/Programming/SeattleLib_v1/semaphore.repy.md +++ b/Programming/SeattleLib_v1/semaphore.repy.md @@ -1,8 +1,8 @@ -# semaphore.repy +# semaphore.repy Although repy already contains a locking method, this module provides a level of abstraction by giving the end users the possibility of using semaphores. Though semaphores are slightly superior to locks, this module is not required to run anything. Using this purely at the programmer's discretion and need. -### Functions +### Functions ``` def semaphore_create(): @@ -15,7 +15,7 @@ def semaphore_destroy(semaphorehandle): ``` Clean up a semaphore that is no longer needed. All currently blocked threads will be unblocked. All future uses of the semaphore will fail. - Notes: + Notes: * semaphorehandle is the semaphore handle to destroy * Returns True if it cleaned up the semaphore handle, False if the handle was already cleaned up. @@ -26,7 +26,7 @@ def semaphore_up(semaphorehandle): ``` Increment a sempahore (possibly unblocking a thread) - Notes: + Notes: * semaphorehandle is the semaphore handle generated by the create method. * Raises ValueError if the semaphorehandle is invalid. @@ -37,11 +37,11 @@ def semaphore_down(semaphorehandle): ``` Decrement a sempahore (possibly blocking this thread) - Notes: + Notes: * semaphorehandle is the semaphore handle generated by the create method. * Raises ValueError if the semaphorehandle is invalid. -### Includes +### Includes -[wiki:SeattleLib/uniqueid.repy] \ No newline at end of file +[uniqueid.repy](uniqueid.repy.md) diff --git a/Programming/SeattleLib_v1/signeddata.repy.md b/Programming/SeattleLib_v1/signeddata.repy.md index 9ecaf2d25..da73d74b7 100644 --- a/Programming/SeattleLib_v1/signeddata.repy.md +++ b/Programming/SeattleLib_v1/signeddata.repy.md @@ -1,20 +1,20 @@ -# signeddata.repy +# signeddata.repy This modules allows data to be signed in order to prevent attacks. By signing the data, we can create signatures, which greatly increase security since data signatures can then be checked at the start and at the end of the transfer. We are trying to prevent four main types of attacks. They are described below. -**Replay attack**: When someone provides information you signed before to try to get you to perform an old action again. For example, A sends messages to the node manager to provide a VM to B (B intercepts this traffic). Later A acquires the VM again. B should not be able to replay the messages A sent to the node manager to have the VM transferred to B again. +**Replay attack**: When someone provides information you signed before to try to get you to perform an old action again. For example, A sends messages to the node manager to provide a VM to B (B intercepts this traffic). Later A acquires the VM again. B should not be able to replay the messages A sent to the node manager to have the VM transferred to B again. -**Freeze attack**: When an attacker can act as a man-in-the-middle and provide stale information to an attacker. For example, B can intercept all traffic between the node manager and A. If C makes a change on the node manager, then B should not be able to prevent A from seeing the change (at least within some time bound). +**Freeze attack**: When an attacker can act as a man-in-the-middle and provide stale information to an attacker. For example, B can intercept all traffic between the node manager and A. If C makes a change on the node manager, then B should not be able to prevent A from seeing the change (at least within some time bound). -**Out of sequence attack**: When someone can skip sending some messages but deliver others. For example, A wants to stop the current program, upload a new copy of the program, and start the program again. It should be possible for A to specify that these actions must be performed in order and without skipping any of the prior actions (regardless of failures, etc.). +**Out of sequence attack**: When someone can skip sending some messages but deliver others. For example, A wants to stop the current program, upload a new copy of the program, and start the program again. It should be possible for A to specify that these actions must be performed in order and without skipping any of the prior actions (regardless of failures, etc.). -**Misdelivery attack**: Messages should only be acted upon by the nodes that the user intended. A malicious party should not be able to "misdeliver" a message and have a different node perform the action. +**Misdelivery attack**: Messages should only be acted upon by the nodes that the user intended. A malicious party should not be able to "misdeliver" a message and have a different node perform the action. -### Functions +### Functions ``` def signeddata_is_valid_expirationtime(expirationtime): @@ -26,14 +26,14 @@ def signeddata_is_valid_timestamp(timestamp): ``` def signeddata_is_valid_sequencenumber(sequencenumber): ``` - Checks if the sequence numbers are valid. They must be in the form 'tag:num' where tag doesn't contain ':',' + Checks if the sequence numbers are valid. They must be in the form 'tag:num' where tag doesn't contain ':',' n', or '!' # and num is a number. ``` def signeddata_is_valid_destination(destination): ``` - Destination is an "opaque string" or None. Should not contain a '!' or ' + Destination is an "opaque string" or None. Should not contain a '!' or ' n'. ``` @@ -83,7 +83,7 @@ def signeddata_shouldtrust(oldsigneddata, newsigneddata, publickey=None, oldsign Checks to see if the data itself can be trusted. -### Usage +### Usage ``` # get some signed data @@ -110,11 +110,11 @@ def signeddata_shouldtrust(oldsigneddata, newsigneddata, publickey=None, oldsign ``` -### Includes +### Includes -[wiki:SeattleLib/sha.repy] +[wiki:SeattleLib/sha.repy](sha.repy.md) -[wiki:SeattleLib/rsa.repy] +[wiki:SeattleLib/rsa.repy](rsa.repy.md) -[wiki:SeattleLib/time.repy] +[wiki:SeattleLib/time.repy](time.repy.md) diff --git a/Programming/SeattleLib_v1/sshkey.repy.md b/Programming/SeattleLib_v1/sshkey.repy.md index 74aff69d0..adbcb3a18 100644 --- a/Programming/SeattleLib_v1/sshkey.repy.md +++ b/Programming/SeattleLib_v1/sshkey.repy.md @@ -1,10 +1,10 @@ -# sshkey.repy +# sshkey.repy sshkey.repy provides an utility for reading public and private keys. This module is used whenever a sshkey needs to be decrypted, and helps the client by providing a level of abstraction in reading these keys. This module is capable of reading private keys encrypted with DES3 as well, in addition to the keys encrypted by [wiki:SeattleLib/rsa.repy]. -### Functions +### Functions ``` def close(self): @@ -17,7 +17,7 @@ def read(self, n = -1): ``` Read at most size bytes from the file (less if the read hits EOF before obtaining size bytes). - Notes: + Notes: * If the size argument is negative or omitted, read all data until EOF is reached. The bytes are returned as a string object. An empty string is returned when EOF is encountered immediately. * Raises ValueError if I/O operation on closed file. @@ -29,7 +29,7 @@ def sshkey_file_to_privatekey(filename, password=None): ``` Reads a ssh private key file and returns the key in a format suitable to used by rsa.repy. - Notes: + Notes: * filename is the name of the file containing the ssh-rsa private key. Key should be either unencrypted or encrypted with DES3. * password is the password used to encrypt the ssh-rsa private key. @@ -51,7 +51,7 @@ def sshkey_file_to_publickey(filename): ``` Reads a ssh public key file and returns the key in a format suitable to used by rsa.repy. - Notes: + Notes: * filename is the name of the file containing the ssh-rsa publickey. * Raises sshkey_SSHException if private key was unable to be decoded. This could happen for any number of reasons and the only quick fix is to generate a new key that is supported. This could happen if the file contains a dss key. @@ -60,7 +60,7 @@ def sshkey_file_to_publickey(filename): * Returns a publickey dictionary in the format used by the rsa.repy module: {'n': 1.., 'e': 6..} with the keys 'n' and 'e'. -### Usage +### Usage ``` @@ -98,6 +98,6 @@ def sshkey_file_to_publickey(filename): assert(publickey['e'] == e) ``` -### Includes +### Includes -[wiki:SeattleLib/sshkey_paramiko.repy] \ No newline at end of file +[sshkey_paramiko.repy](sshkey_paramiko.repy.md) diff --git a/Programming/SeattleLib_v1/time.repy.md b/Programming/SeattleLib_v1/time.repy.md index ebc196d2c..5eda2b0e4 100644 --- a/Programming/SeattleLib_v1/time.repy.md +++ b/Programming/SeattleLib_v1/time.repy.md @@ -1,9 +1,11 @@ -# time.repy +# time.repy A include file to tie the time interface together. Contains no functions. -### Includes +### Includes -[wiki:SeattleLib/ntp_time.repy] +[wiki:SeattleLib/ntp_time.repy](ntp_time.repy.md) -[wiki:SeattleLib/tcp_time.repy] \ No newline at end of file +[wiki:SeattleLib/tcp_time.repy](tcp_time.repy.md) + +[Back to Time](Time.md) diff --git a/Programming/SeattleLib_v1/uniqueid.repy.md b/Programming/SeattleLib_v1/uniqueid.repy.md index 66d534b11..8287c7af7 100644 --- a/Programming/SeattleLib_v1/uniqueid.repy.md +++ b/Programming/SeattleLib_v1/uniqueid.repy.md @@ -1,20 +1,20 @@ -# uniqueid.repy +# uniqueid.repy This is a simple module which provides an unique integer id for each function call. This exists to reduce redundancy in other libraries. -This service can be utilized directly, but is used in contexts like [wiki:SeattleLib/parallelize.repy]. +This service can be utilized directly, but is used in contexts like [parallelize.repy](parallelize.repy.md). NOTE: This will give unique ids PER FILE. If you have multiple python modules that include this, they will have the potential to generate the same ID. -### Functions +### Functions ``` def uniqueid_getid(): ``` Return a unique ID in a threadsafe way -### Usage +### Usage ``` request_id = uniqueid_getid(); -``` \ No newline at end of file +``` diff --git a/Programming/SeattleLib_v1/xmlrpc_common.repy.md b/Programming/SeattleLib_v1/xmlrpc_common.repy.md index 41cd7374a..c87e67dfa 100644 --- a/Programming/SeattleLib_v1/xmlrpc_common.repy.md +++ b/Programming/SeattleLib_v1/xmlrpc_common.repy.md @@ -1,4 +1,4 @@ -# xmlrpc_common.repy +# xmlrpc_common.repy This module helps to implement the client-side XML-RPC protocol. See http://en.wikipedia.org/wiki/XML-RPC for more details. @@ -6,13 +6,13 @@ A programmer may use this module to initiate a communication between the client Note that this module is primarily used by [wiki:SeattleLib/xmlrpc_client.repy], which programmers should include instead of this file. This file contains many functions which allow the XML-RPC protocol to be used in Python and is primary used to build the strings necessary to implement XML-RPC. -### Functions +### Functions ``` def xmlrpc_common_call2xml(method_name, params): ``` Build a XML-RPC method call to send to a XML-RPC server. - Notes: + Notes: * params is the sequence type of XML-RPC parameters. A dictionary may also be passed here, but the keys are ignored. * Returns the XML-RPC method call string. @@ -22,7 +22,7 @@ def xmlrpc_common_response2xml(param): ``` Build a XML-RPC method response to send to a XML-RPC client. This is the XML document that represents the return values or fault from a XML-RPC call. - Notes: + Notes: * param is the value that is returned (aka the response). * The XML-RPC method response string. @@ -32,7 +32,7 @@ def xmlrpc_common_fault2xml(message, code): ``` Build a XML-RPC fault response to send to a XML-RPC client. A fault response can occur from a server failure, an incorrectly generated XML request, or bad program arguments. - Notes: + Notes: * message is the string describing the fault. * code is the integer code associated with the fault. @@ -43,7 +43,7 @@ def xmlrpc_common_call2python(xml): ``` Convert a XML-RPC method call to its Python equivalent. The request from a XML-RPC client is parsed into native Python types so that the server may use the data to execute a method, as appropriate. - Notes: + Notes: * xml is the XML-RPC string to be convert. * Raises xmlrpc_common_XMLParseError on a XML-RPC structural parse error. @@ -55,21 +55,21 @@ def xmlrpc_common_response2python(xml): ``` Convert a XML-RPC method response to its Python equivalent. The response from a XML-RPC server is parsed into native Python types so that the client may use the data as appropriate. - Notes: + Notes: * xml is the XML-RPC string to be convert. * Raises xmlrpc_common_XMLParseError on a XML-RPC structural parse error. * Raises xmlparse_XMLParseError on other XML parse errors. * Returns the method results or a xmlrpc_common_Fault on reading a fault. -### Example Usage +### Example Usage ``` client = xmlrpc_client_Client("http://phpxmlrpc.sourceforge.net/server.php") print client.send_request("examples.getStateName", (1,)) ``` -### Includes -[wiki:SeattleLibbase64.repy] +### Includes +[base64.repy](base64.repy.md) -[wiki:SeattleLib/xmlparse.repy] +[xmlparse.repy](xmlparse.repy.md) diff --git a/Scripts/README.md b/Scripts/README.md new file mode 100644 index 000000000..8f317e0cc --- /dev/null +++ b/Scripts/README.md @@ -0,0 +1,67 @@ +# `auto_grader.py` +> **Automated Grading for Repy V2 Assignments** + +`auto_grader.py` is a Python-based tool designed to streamline the grading +process for defense and attack programs written in [Repy +V2](https://github.com/SeattleTestbed/repy_v2). By automatically running each +attack against every defense, it produces comprehensive results in a CSV format. + +## Description +With `auto_grader.py`, instructors can effortlessly assess the efficacy of +students' defense mechanisms in the face of potential attacks. For every attack +that succeeds in compromising a defense, the resulting CSV will register a `1`; +otherwise, it will display a `0`. An attack is considered successful if it +generates an output or raises an error, denoting the failure of the defense +layer. It also handles timeouts, ensuring that the script does not hang in the +event of an infinite loop. + +## Prerequisites +- Python 2.7 installed on your machine. +- Repy V2 environment setup. +- Copy the required files (mentioned below) in the script's directory + - `repy.py` + - `restrictions.default` + - `wrapper.r2py` + - `encasementlib.r2py` + +## Usage +```bash +python auto_grader.py defense_folder_path attack_folder_path temp_target_folder_path +``` +where: +- `defense_folder_path` is the path to the folder containing defense programs. +- `attack_folder_path` is the path to the folder containing attack programs. +- `temp_target_folder_path` is the path to the temporary target folder. + +## Naming Conventions +- Attack programs should be named as: "`[studentid]_attackcase[number].r2py`". +- Defense programs should start with the name + "`reference_monitor_[studentid].r2py`". + +For example, if the student id is `abc123`, the defense program should be named +as `reference_monitor_abc123.r2py` and the attack program should be named as +`abc123_attackcase1.r2py`, `abc123_attackcase2.r2py`, etc. + +## Output +Two CSV files are generated: +1. `All_Attacks_matrix.csv`: Contains the result of every attack program against + each defense. +2. `All_Students_matrix.csv`: Indicates which students successfully attacked a + defense. + +## Notes +- Students are instructed to generate output or raise an error in their attack + program only when they successfully compromise the security layer. +- Ensure the correct environment, naming conventions, and directory structures + are adhered to for successful script execution. + +## Contributing +For modifications, improvements, or any issues, please open a pull request or +issue. + +## Credits +`auto_grader.py` is the brainchild of +[@Hooshangi](https://github.com/Hooshangi). + +For further details, potential contributions, or to view the code, visit +[Hooshangi/Grading-script](https://github.com/Hooshangi/Grading-script). diff --git a/Scripts/auto_grader.py b/Scripts/auto_grader.py new file mode 100644 index 000000000..c0f0dfc35 --- /dev/null +++ b/Scripts/auto_grader.py @@ -0,0 +1,235 @@ +""" + + auto_grader.py + + + Module which takes all defense programs and attack programs in Repy V2 and creates + csv file describing test result of every attack program against each defense program. + Under the attack program column, it shows one if attack was able to compromise security layer + and 0 otherwise. + Students submitting attack and defense programs are instructed to log output or raise error + in their attack program only when the security layer gets failed. In this program we check which + attack program is producing output or error. If so that means the attack is successful and that + security layer is compromised. + + + -All defense programs should be in a folder and all attack programs in another folder. + auto_grader, repy.py ,restrictions.default, wrapper.r2py, encasementlib.r2py. + -Create a temporary target folder inside the directory. + + python auto_grader.py defense_folder_path attack_folder_path temp_target_folder_path + +""" + + + + +import os +import glob +import subprocess +import shutil +import csv +import time +import signal # using SIGKILL +import sys + + +path_DefenseFolder = sys.argv[1] +path_AttackFolder = sys.argv[2] + + +# Paths to temp folder where the attack is run +path_TempFolder = sys.argv[3] + +#part of filename to look for +attack_ext = "*.r2py" #extension of attack programs will be r2py +def_ext = "reference*" #students are instructed to name their defense program starting with reference + + + + +def get_student_ID(attack_fnlist): + ''' Given a list of the total attack files, get a list of all of the + student IDs. This assumes that the file names contains _ + ''' + + studentset = set() + for attackfn in attack_fnlist: + # I assume everything before the first _ is the student ID. I will + # raise an exception for a file like foo.repy... + + if len(attackfn.split('_')) == 0: + raise ValueError('File name "'+attackfn+'" should contain an underscore!') + + studentname = attackfn.split('_')[0] + studentset.add(studentname) + + return sorted(list(studentset)) + + +def get_student_attack_code(studentname, attack_fnlist): + ''' This will return only the attack code from a specific student.''' + + thisstudentattacks = [] + for thisattack in attack_fnlist: + # need the underscore to stop bob from matching bobby's tests. + if thisattack.startswith(studentname+'_'): + thisstudentattacks.append(thisattack) + + + return thisstudentattacks + + +def check_if_student_attacks_succeed(student_attackfn, defensefn): + ''' Returns a list of any code for a student that produce stderr or stdout. + An empty list is returned if none were successful + otherwise''' + + successfulattacks = [] + + for attackfn in student_attackfn: + if did_this_attack_succeed(attackfn, defensefn): + successfulattacks.append(attackfn) + + return successfulattacks + + + + +def did_this_attack_succeed(attackFilename, defenseFilename): + ''' Returns True if the attack produces stderr or stdout, + False otherwise + ''' + + timeout=30 + + os.mkdir(path_TempFolder) # make a temp folder + os.chdir(path_TempFolder) # cd to temp folder at this point + + shutil.copy(path_DefenseFolder + '/' + defenseFilename, path_TempFolder + '/' + defenseFilename) + shutil.copy(path_AttackFolder + '/' + attackFilename, path_TempFolder + '/' + attackFilename) + shutil.copy('../wrapper.r2py', path_TempFolder + '/wrapper.r2py') + start = time.time() + + pobj = subprocess.Popen( + ['python', '../repy.py', '--stop=Repy_stop_this', '../restrictions.default', '../encasementlib.r2py', defenseFilename, + attackFilename], stdout=subprocess.PIPE, stderr=subprocess.PIPE) + + # NOT A BUG: Note that this will not get all of the stdout or stderr because we are polling for completion. + # Substantial output may cause the program to block, which is a very bad thing... in most cases. + # Since output / errput is failure and timeout is failure, we're actually okay with it here. + while pobj.poll() is None: + time.sleep(0.1) + now = time.time() + if now - start > timeout: + # Signal for the repyVM to stop (due to the --stop option) + file("Repy_stop_this","w").close() + # wait for it to stop... + pobj.wait() + stdout = "timeout" + stderr = "timeout" + break + else: + (stdout, stderr) = pobj.communicate() + + + os.chdir(path_AttackFolder) # go back to attack folder + shutil.rmtree(path_TempFolder) #remove the temp directory + + + if stdout != '' or stderr !='': + return True + else: + return False + + + +def row_builder(success_list,original_list,defenseFilename): + + num_success=len(success_list) + + matrix_row=[0]*(len(original_list)) + matrix_row[0]=defenseFilename + + for element in range(num_success): + + for row_num in range(len(original_list)): + if original_list[row_num]==success_list[element]: + matrix_row[row_num]=1 # if an attack is successful, put a 1 in the matrix + + return matrix_row + + + + + + +def main(): + + # If the temp folder is left over from last time, remove it. + if os.path.exists(path_TempFolder): + shutil.rmtree(path_TempFolder) + + os.chdir(path_DefenseFolder) # cd to defense monitor folder + defense_fnlist =glob.glob(def_ext) + + os.chdir(path_AttackFolder) # cd to attack folder + attack_fnlist=glob.glob(attack_ext) + + studentIDlist = get_student_ID(attack_fnlist) + print 'number of students in the course', len(studentIDlist) + + header_attackmatrix=attack_fnlist + header_attackmatrix.insert(0,'All attack files-->') + + header_studentmatrix=studentIDlist + header_studentmatrix.insert(0,'All students -->') + + resultFile1 = open("All_Attacks_matrix.csv",'wb') + wr_allattacks = csv.writer(resultFile1) + wr_allattacks.writerow(header_attackmatrix) + + resultFile2 = open("All_Students_matrix.csv",'wb') + wr_students = csv.writer(resultFile2) + wr_students.writerow(header_studentmatrix) + + + for defenseFilename in defense_fnlist: + collection_successful_attacks=list() + collection_successful_students=list() + + print 'This is defense file --->', defenseFilename + + for attackingstudent in studentIDlist: + + # get just this student's attacks + student_attackfns = get_student_attack_code(attackingstudent,attack_fnlist) + successfulattacks = check_if_student_attacks_succeed(student_attackfns,defenseFilename) + + + if successfulattacks!=[]: + + print defenseFilename,'---attacked by--- ', attackingstudent,'----->', successfulattacks + collection_successful_attacks=collection_successful_attacks+successfulattacks + collection_successful_students.append(attackingstudent) + + #print 'successful attacks',collection_successful_attacks + row_all_attacks=row_builder(collection_successful_attacks,attack_fnlist,defenseFilename) + row_students=row_builder(collection_successful_students,studentIDlist,defenseFilename) + + wr_allattacks.writerow(row_all_attacks) + wr_students.writerow(row_students) + + print 'row completed with students',row_students + + + + + +if __name__ == "__main__": + main() + + + + + diff --git a/UnderstandingSeattle/SeattleShellBackend.md b/UnderstandingSeattle/SeattleShellBackend.md index c1aab34a3..6ce931474 100644 --- a/UnderstandingSeattle/SeattleShellBackend.md +++ b/UnderstandingSeattle/SeattleShellBackend.md @@ -1,10 +1,10 @@ -Seash (also known as Seattle Shell) is the command line interface for interacting with Seattle VMs. This will describe how seash is designed. If you are a user interested in learning how to use Seash, please go to the [SeattleShell main Seattle Shell page]. +Seash (also known as Seattle Shell) is the command line interface for interacting with Seattle VMs. This will describe how seash is designed. If you are a user interested in learning how to use Seash, please go to the [SeattleShell main Seattle Shell page](SeattleShell.md). - -## Command Dictionary + +## Command Dictionary ---- The command dictionary contains command nodes that determine how seash responds to various input. Each command node represents a recognized command keyword, and may have child command nodes that represent keywords that should come after that command. Seash will use this dictionary to check if an input string matches a path through the command dictionary, and if it does, executes the command at the terminating node. @@ -48,16 +48,16 @@ command_dictionary = { ... } ``` - -## Defining New Modules + +## Defining New Modules ---- Seash has support for adding additional commands via modules that are imported on startup. These modules should be placed in a subdirectory within the /modules/ folder. These modules are standard python packages. - -### Creating a Module + +### Creating a Module Defining a module is relatively straightforward. First, create the ```__init__.py``` file that contains the ```moduledata``` dictionary in the module namespace that is loaded by the module importer. It should look like the following: ```python @@ -70,9 +70,9 @@ moduledata = { 'input_preprocessor': input_preprocessor, } ``` - -#### The Command Dictionary + +#### The Command Dictionary The command_dict defines all the commands that are to be part of the module. These command_dicts should be specified in a similar manner as described in the command dictionary section above. @@ -90,9 +90,9 @@ command_dict = { 'summary':'Display the latitude & longitude of the node', 'help_text':"""...""",} ``` - -#### Module Level Documentation + +#### Module Level Documentation You must also have module-level documentation. ```python @@ -105,9 +105,9 @@ Clearinghouse, use the 'browse' command, and then in any group, run either 'show or 'show coordinates'. """ ``` - -#### Automatic Updating + +#### Automatic Updating You must also specify an update URL to where your module can be found. This will be used with the updater to automatically update the module. To disable this feature, set the URL to ```None```. Note, this is not functional yet. All modules should not have this set until then. ```python # This is already set in the moduledata above so this is no longer needed. @@ -116,9 +116,9 @@ You must also specify an update URL to where your module can be found. This wil moduledata['url'] = "http://update.url/" ``` - -#### Input Preprocessor + +#### Input Preprocessor The input preprocessor is available for modules to tap into raw input from the command line to perform initial preprocessing. This is used especially in the variables module. This preprocessor will be given one string as input, and must return one string representing the results of the preprocessing. This example is a simple example that replaces every instance of $USERPORT with a user's port. @@ -135,4 +135,4 @@ def input_preprocessor(user_input): else: retstr += user_input return retstr -``` \ No newline at end of file +```