Assignment 1 - COMP9322 CSE @ UNSW (17s1) Assignment One Developing and Consuming SOAP-Based Services Due Dates: 2nd April Sunday 11:00PM 15 marks total (13 marks for the assignment solution, 2 marks for peer assessment) This is an individual assignment - no group activity allowed. Overview The main learning outcomes of this assignment are as follows design and implementation of SOAP-based services, consume SOAP-based services to build an application, create and manage a portable deployment environment, be able to deploy, test and assess services developed by others In this assignment, you will one SOAP-based service with a number of operations. You will also build a fully functioning client Web application that utilises the SOAP service. Then, build and deploy both the service and the client application on Docker. You will submit both the source code (cleaned Eclipse project folders) and a Docker image file. Assignment One has four parts. 15 marks in total. Part I (7 marks) : Implementing the Stocket Market Data Related Services Part II (4 marks): Market Data Service Client Web App Part III (2 marks): Docker Deployment and Management Part IV (2 marks): Peer Assessment Participation Test Data and FAQ : Listing of Test Data and FAQ Assignment Submission Guide Background Note: The basic skeleton of the services described here is introduced in Lab01 and Lab02. We assume that you have completed both labs successfully. The original specification of these services were developed as part of Adage project — ad-hoc data analysis platform, mainly fine-turned for financial data. The Web services deal with stock market data which is formatted according a schema (more details about the data set is below). In this assignment, we are trying to implement some of the services there ourselves. We first explain the high-level idea behind the services. To start with, there is a data importing operation. You will take an input file from the Internet (i.e., read from a URL) which contains market events data, perform some filtering on its contents, then write an output file into a data repository (in this assignment, this repository is just your file system). These files are given unique IDs (by you), named EventSet ID. These IDs are then used like access handles for the corresponding files. The users of your services use the IDs to perform other 'operations' on the files. For example, one can download the files, sort the content by some criteria (see Figure 1). Figure 1: Workflow of Market Data Import/Download Service The Market Data files shown in the upper left corner of Figure 1 are a special type of Comma Separated Values (CSV). It stores a record of the events that have occurred at some financial market in the world. For the purpose of this assignment, these events may be trades or quotes indicated by the participants of the financial market. See the Market Data Specification for details about the file format. Part I: Implementing the Stocket Market Data Related Services Out of all the possible operations in this scenario, we will implement the following four: importMarketData() — importing Market Data into the system downloadFile() — Download Market Data as a file convertMarketData() — Convert the dollar amount in the source content into another currency yahooExchangeRate() — SOAP-Wrapper of an existing currency convert service available on-line The above operations will be implemented as a single SOAP service. That is, you will design one WSDL file that contains these operations. You can look at the TopDownSimpleService.wsdl file given in the labs as your starting point. Most importanlty, keep in mind that the Web service operations must work with Market Data files and you should know the specification of the Market Data file. Import Market Data operation: importMarketData() A Market Data file may contain a mixture of trades or quotes relating to several financial instruments. An example of financial instrument is BHP shares traded on the Australian Stock Exchange. The Import Market Data operation is designed to filter a Market Data file, so a new Market Data file is created with only those events from instruments that a user is interested in. For example, a user can use the Import Market Data operation to extract the last 3 months of BHP trades given an input Market Data file having trades and quotes relating to the top 200 companies on the Australian Stock Exchange for the last 3 years. The importMarketData() operation reads a Market Data file that exists outside of the system, filters the content according to the given parameters, and produces another Market Data file to store it inside the system. The filter conditions are specified by the input parameters. It accepts the following input parameters. Parameter Description sec The security code of the financial instrument (i.e., stock). All data rows in the Market Data file having a security code that matches this parameter should be included in the output Market Data file. startDate Rows that have a timestamp on or after (i.e., inclusive) this parameter and also satisfy the condition for endDate, are included in the output Market Data file. It is represented as a string with DD-MM-YYYY format. endDate Rows that have a timestamp on or before (i.e., inclusive) this parameter and also satisfy the condition for startDate, are included in the output Market Data file. It is represented as a string with DD-MM-YYYY format. dataSourceURL This is a http:// URL that points to the input Market Data file. You can use the java.net.URL class to read the Market Data file referenced by this parameter. This operation has a single output parameter described below. Parameter Description eventSetId An opaque "handle" or reference that refers to the output Market Data file (also called an event set: because it stores an ordered set of events). This parameter can be any identifying string, it needs to be meaningful only within the context of the Web services within your system. Sample SOAP request (Body):
WSSS
08-04-2002
09-04-2002
http://adage.cse.unsw.edu.au:8080/MarketData03.csv
Sample SOAP response (Body):
tsb-9b7e45ad
Notes: Essentially, this operation filters the input Market Data. A data row passes through the filter if and only if it matches the security code AND has a date between the start date and time (inclusive) AND end date and time (inclusive). If a Web browser is used to save the contents from the dataSourceURL, the downloaded contents would be Market Data, not compressed data or a ZIP file. Even though eventSetId String is opaque, it should still only contain printable ASCII characters. No control characters nor binary data should be used. For the Market Data repository (private), you may create files or directories on the local file system. The preferred place to use is the directory specified by the Java system property java.io.tmpdir. Alternatively you may call java.io.File.createTempFile(String, String) method to create new files. Important!The value of the eventSetId output parameter must be valid from the first time your Web service is operational (i.e. the Tomcat server is first started) and should persist. This means it must still be valid even if we stop and restart the Tomcat server running your Web service. The eventSetIds generated during a single running session of Tomcat should be available throughout the session. However, you can ignore the statement about having to persist eventSetIDs even after a tomcat shutdown. This won't be that easy to do in our current setup of Docker, so we will ignore it. Download File operation: downloadFile() Since the Market Data file output from Import Market Data operation is created on the local file system of the server, the users will need to specifically request to download the files over the Internet. Keep in mind that anybody with a valid eventSetId can download the corresponding Market Data. The downloadFile() operation makes the imported Market Data accessible to others and it accepts the following parameters as input. Parameter Description eventSetId An opaque "handle" or reference that is the output of the importMarketData operation. If an invalid value is passed (i.e. not an output of the importMarketData operation) then this operation should produce the appropriate SOAP fault. fileType The format of the file. The user can request the Market Data to be in CSV (i.e., as is), or in XML, or in HTML. Therefore, the valid input values for this parameters are: CSV, XML, HTML. Anything else should throw an appropriate SOAP Fault. It has a single output parameter described below. Parameter Description dataURL This is a http:// URL that points to the Market Data file referenced by the eventSetId input parameter. A user can download the Market Data file contents using this URL (e.g. by navigating to this URL using a Web browser). The downloadFile() response type is updated. See the course noticeboard for details. Sample SOAP request (Body):
tsb-9b7e45ad
CSV
Sample SOAP response (Body):
http://ec2-54-252-161-7.ap-southeast-2.compute.amazonaws.com:8080/tsb-9b7e45ad-now.csv
Notes: If a Web browser is used to save the contents from the dataURL, the downloaded content must be Market Data, not compressed data or a ZIP file. You may decide the schema for XML and HTML files. The files do not need to be verified against the schema though. We will just take whatever format you decide to use. Anything simple, but consistent across your service is good. Currency Exchange Rage operation: yahooExchangeRate() For this service, you are going to 'wrap' an existing Web-based currency convert service and create our own SOAP-based service interface for it. The source service: We will use a currency convert/rate service from Yahoo Finance Web Site. Examine the following URL:
http://finance.yahoo.com/d/quotes.csv?e=.csv&f=sl1d1t1&s=USDAUD=X
If you follow the above URL, it will give you a small CSV file containing something like:
"USDAUD=X",1.3020,"3/16/2017","5:03am"
The content represents the exchange rate from USD to AUD (as of 6th March 2016, 5:03am). You should try the URL with different query parameter values to understand its behaviour more. But mainly we are interested in the query parameter 's'. The parameter 's' seems to have the following format:
s=AAABBB=X
Here, AAA is a three character long currency code of the base and BBB is the currency code of the target, followed by '=X'. For implementingyahooExchangeRate(), you will be given a sample SOAP input and output. From there, you will design your own operation interface. 'Wrapping' the Yahoo Finance currency rate service as a SOAP-service would mean that you will receive a SOAP request, you will map the request to the URL parameters of Yahoo Finance currency rate service. Read the content of the csv file and map it to a SOAP response you are expected to return. Here is an example of SOAP request/response for the operation. SOAP request (Body) - values are given as examples:
EUR
USD
SOAP response (Body) - values are given as examples:
1.1071
18-08-2015
Convert Market Data operation: convertMarketData(): We assume all price information (i.e., Price, Bid Price, Ask Price) in the Market Data files to be in Australian Dollars (AUD). This service takes a Market Data file and convert all price information present in the file to another currency. For such conversion, we will use the Yahoo currency rate service you have implemented previously. The operation convertMarketData() has the following SOAP message details: Sample SOAP request (Body):
tsb-9b7e45ad
USD
Sample SOAP response (Body):
rsy-1c7e67tg
The new eventSetId returned by this operation should contain exactly the same information as the original file, except for the new/currency converted price information. The converted price information should have the target currency code as pre-fix. To simplify, we assume that the conversion is always based on "today's rate". We do not have to worry about the historical rates to match the event date for this exercise. For example, let's say the Market Data file tsb-9b7e45ad contains:
#RIC,Date[G],Time[G],GMT Offset,Type,Price,Volume,Bid Price,Bid Size,Ask Price,Ask Size
ABCD,26-JUN-2001,12:54:50.712,+2,Trade,57.8,,,,,
ABCD,27-JUN-2001,07:21:25.999,+2,Trade,58.35,,,,,
...
The new Market Data file rsy-1c7e67tg should contain:
#RIC,Date[G],Time[G],GMT Offset,Type,Price,Volume,Bid Price,Bid Size,Ask Price,Ask Size
ABCD,26-JUN-2001,12:54:50.712,+2,Trade,USD53.8,,,,,
ABCD,27-JUN-2001,07:21:25.999,+2,Trade,USD54.31,,,,,
...
One can always use the downloadFile() from ImportDownloadServices to get the content of this file. Note: Let us assume that once the price information is converted from the default Australian dollar to another currency, there should be no more currency conversion applied to the data. That is, the service should check if the input eventSetID contains price information which is already converted. For example, for a SOAP request
rsy-1c7e67tg
JPY
The operation should return a SOAP fault similar to:
The file already contains converted prices
Handling SOAP Faults: In Lab02, you have seen how to declare and implement SOAP faults in WSDL. In the assignment, a fair portion of the marking and testing will be dedicated to assessingn your SAOP faults solution. You need to define appropriate fault message types for ALL operations, and if you detect an error condition (e.g. end date before start date), raise an appropriate faultType and faultMessage indicating the problem. Some of the possible fault types are: InvalidEventSetId - the input eventSetId is not known on the server repository. InvalidURL - the input URL is either in the wrong format, or not reachable InvalidSECCode - the input sec is not known on the server repository, or not in the expected format InvalidDates - the end date must be before the start date InvalidFileType - the input fileType is not one of the expected values InvalidCurrencyCode - the input currency code is not one of the expected values ProgramError - any other error On validating things (hence generating SOAP Faults): When validating the input, you may want to consider two kinds of checks: (i) when an input is given in a wrong format/type, (ii) a given input does not match what you have. I take the SEC code as an example. SEC code: a SEC code should in upper case, alphabetics only and 3-4 characters in length. You should throws an InvalidSECCode fault, if the input does not satisfy this syntax. You should also throw an InvalidSECCode fault when the SEC code does not match any existing code in the given Market Data file. You should give an explanation of the fault properly always in the fault message. Try to apply this principle in all inputs. Should there be a case where an import operation will generate an eventSetId whose content is empty? Interceptor Processing for SOAP messages Similar to what is shown in the HelloCXFIntercept project (from Week 3 Sample Codes), into your service, you will build a custom logging function. In this logging function, you will do the following: Any incoming SOAP messsage to your service, you will inspect the header section, and print (to standard out) the fully-qualitifed name of all header elements and their values. If you do not see any header elements, print a message stating so. For example, if you see the following SOAP header elements in a message,
soacourse-key-pass
soacourse
note: soapenv:mustUnderstand is not necessary.
soacourse-key-pass
soacourse
You should print:
A message received with the following headers:
http://soacourse.cse.unsw.edu.au:api-key
http://soacourse.cse.unsw.edu.au:api-keyvalue has soacourse-key-pass
http://soacourse.cse.unsw.edu.au:api-keyuser has soacourse
Part II: Market Data Client Application In this part of the assignment, you will develop a simple Web application that utilises the Market Data services. The main exercise point in this part of the assignment is to let the client application 'compose' the available services manually (e.g., chaining up three services at one go). Follow the instructions in Lab02 to setup a SOAP client Web application using Apache CXF. As a general rule, your client application should behave gracefully when it is dealing with the Web services - meaning when the SOAP responses from the SOAP services contain faults, your application should display the faults properly (as if you are displaying a normal response), rather than showing raw messages (e.g., raw server error message). Process the fault message, read error code, error text and print them in HTML in a user friendly manner. The Market Data Client application is a Web-based application and has the following functions for the (human) users. The user can import Market Data into the system and obtain an eventSetId. Any eventSetId obtained during a session will be clearly listed and visible at the top of the page. This list should be updated every time a new eventSetId is generated. Note: you do not need to manage sessions here. The list will be only valid during a single session. The user can run chain multiple services at one go. Two options are offered. importMarketData, followed by convertMarketData, followed by downloadFile importMarketData, followed by downloadFile The following mock-up screens suggest possible layout of the client app. Mockup screen You can be as creative as you'd like. Part III: Docker Deployment and Management You will deploy the solutions from Part 1 and Part 2 (i.e., two WAR files) to a tomcat container in Docker, using the same procedure specified in the labs. You will then save and export this container as a zipped, TAR file and submit for assessment. In this part, we will ask you to test this setup to make sure that your solution can be marked correctly. First start the server comp9322srv, then: start the tomcat container; copy and deploy the two WAR files (i.e., one for SOAP service, the other for Client Web app) to the container test both applications and make sure that they are correctly functioning. leave the container running and open another terminal. Now run the following commands to save and export this instance as a Docker image file. Note that these steps are introduced in at the end of Lab04. Replace 'give-your-own-name-here' should be replaced with your own choice for the file name. $ docker commit tomcat8 give-your-own-name-here $ docker save --output give-your-own-name-here.tar give-your-own-name-here:latest To make sure that this image can be successfully loaded and run, do the following (Stop the tomcat container if it is still running). $ docker load --input give-your-own-name-here.tar $ docker run -it --rm --name give-your-own-name-here -p 8888:8080 give-your-own-name-here:latest Test both applications and make sure that they are correctly functioning. You can test this on a new Docker server, other than comp9322srv as well. In the submission instructions, we will explain how to submit this image file. Part IV: Peer Assessment Participation Details to be provided after the submission due date for Part I and II. However, we briefly give you an overview here. As advertised at the start of the session, we will use a peer marking scheme for Assignment 1. You will be allocated two submissions from other students to mark. You will be given a marking criteria to follow to assess the solutions. You are expected to make your own comments about the solution along the way. At the end of the marking process, you will get a mark from the students you assessed to indicate how ``well'' you have performed as an assessor. This process is supported by a system developed at University of Tasmania, Smart Service Systems Lab. In the submission instructions, we will explain how to submit your solution to the system and participate in the peer marking process. Test Data and FAQ (hints on Part 1) Sample test data is available from: FinDataSimple.csv MarketData04.csv MarketData05.csv MarketData06.csv Frequently Asked Questions Q: How do we make the filtered output files available to the Web browser as a URL? This question concerns the Download File operation. What we like you to do is to copy to the tomcat/webapps/ROOT/ directory any file you want to make visible to the Internet. For example, if you have a file output.csv in tomcat/temp directory (i.e. the directory of the system property java.io.tmpdir), you can then copy the file (tomcat/temp/output.csv) to tomcat/webapps/ROOT/output.csv. Then you can access this file using the URL http://localhost:8080/output.csv. Ideally, you would return either the public IP address or host name instead of "localhost" to make the file accessible over the Internet. The above FAQ is not relevant to the spec of downloadFile() after the update. See course noticeboard. Q: Do we need to perform any kind of clean up? For example, delete files after it has been downloaded. Yes, you can clean up. Q: How do we find out at runtime where webapps/ROOT is located? There are a few ways, one is to use the system property catalina.home (similar to how you use java.io.tmpdir or user.home). The catalina.home property will point you to the base location of the Tomcat server. From there you just add the webapps/ROOT bit. Another way (thanks to Liangliang Zheng) is to obtain information from context properties: import javax.servlet.http.HttpServletRequest; import org.apache.axis2.context.MessageContext; ... MessageContext mc = MessageContext.getCurrentMessageContext(); HttpServletRequest req = (HttpServletRequest)mc.getProperty("transport.http.servletRequest"); ... The above FAQ is not relevant to the spec of downloadFile() after the update. See course noticeboard. Q: I need to save some data into a file, where can I create files? You can create your file in the directory specified by the system property java.io.tmpdir. Alternatively, you can create it in webapps/ROOT if it is to be accessible over HTTP. Q: Do I have to filter the data rows in the Market Data file by "DATE" or "DATE and TIME"? DATE only Q: What about time zones? GMT offset? It is quite complicated to take care of time zones and daylight savings in different countries. So for this assignment, we will treat the Date and Time as local date and time (i.e., ignore the GMT offset) Q: Do I have to handle the case when two people try to call my Web service at the same time? No. You do not have to handle concurrency issues for this assignment. Q: Can we use external APIs for this assignment (e.g. CSV readers or writers)? Yes, You can use external APIs or libraries. Make sure that those are managed by pom.xml through Maven dependencies. Q: What happens if I detect multiple errors in my Web service? You only need to detect the first error. After finding the first error, you should raise the appropriate fault. Assignment Submission You will submit the solution in Maven project format using WebCMS3 submission menu (to be opened later), and a docker image file by uploading it onto your CSE account. More detailed instructions will be given when the submission site is open. The marking guide will be released later as well.
