Scenario / testbed
The assignment will be graded based on the groups’ ability to produce useful and correct information within the boundaries of the given time and resources.
You must design a set of experiments to be performed using the testbed provided for the assignment and available services on the web.
The testbed scenario is the case of a major hosting provider for web services that needs to evaluate the transport protocol alternatives for its servers. The data centre of the provider is well provisioned with resources and bandwidth, so one assumption is that any network bottleneck is located at the edge close to the client. The below figure gives a visual presentation of the scenario.
For creating a testbed emulating this scenario, we will provide each group with a virtual machine hosted at the Amazon EC2 server farm in Ireland (eu-west-1). This VM will serve as the client for the web connections. You will need to create a bottleneck on this receiver machine, limiting incoming traffic to the target capacity. The below figure explains how this setup should be.
Consult the course Amazon VM and Transport Protocol FAQ for details on how this can be set up in practise.
Since the virtual machine has built-in capacity limitations based on the instance pricing, we must make sure that we create a bottleneck that is low enough that you don’t end up measuring the token bucket bandwidth-limitation of the virtual machines. You must therefore make sure that the capacity at the bottleneck never exceeds 10Mbps.
The VM will serve as your web client. In order to collect web traffic measurements, we have provided you with two web servers that serves TCP and QUIC respectively:
Use these servers to collect measurement statistics for your evaluation.
For this assignment, we assume that the target of the service provider is to deliver web traffic and that the main purpose of the measurement is to measure the flow completion time, page load time and packet loss rates for the measured traffic. Additionally you are encouraged to look for other appropriate metrics that will aid the evaluation with regard to the protocols you will be evaluating.
You may choose the tools for performing the experiments and analyse the results based on what you are familiar with and what you deem most appropriate for the purpose. The important aspect is that you are able to produce a complete and clear report at the end of the assignment period.
You should evaluate each transport protocol in competition with other flows running the same protocol and in combination with flows running the other protocol you are evaluating (TCP vs. TCP, TCP vs. QUIC and QUIC vs. QUIC).
Take into consideration that the servers might be unevenly loaded over time, since you cannot control who accesses the servers at different times. You need to perform enough tests to make sure that your results are representative of protocol performance and not just artifacts of random competing web traffic.
We encourage you to discuss the challenges and techniques across groups to reduce the overhead in attaining a new field of knowledge. Copying of code, scripts or experimental results, however, will be counted as cheating.
Report
You must write up the results as a technical report of no more than 4 pages in ACM format. It is expected that such a report includes the core elements presented in the lectures under “A systematic approach to performance evaluation”.The results must be based on your own experiments and your own data.
The report is evaluated by writing quality, clarity of presentation, by the trustworthiness and correctness of the results. The evaluation does not consider whether related work (citations of other papers) is included.
Evaluation details
In our evaluation of the reports, we will focus on the following elements:
- Choice of metrics, workloads, system configuration parameters and methodology for the experiments
- Use of statistical sound methods when analysing the data
- Disposition of the available time (ability to collect and present useful information within the boundaries of the available resources)
- Objectivity in defining the work, choosing metrics and workloads, in the analysis and in presenting the results
- Transparency of reporting (exposure of assumptions and limitations to the reader)
- Clarity of presentation
Bonus elements:
- Analysis of metrics, beyond the core metrics listed above, that helps illustrate the qualities of the different transport protocols.
- Analysis of experiments using other external websites to measure against (for instance Google sites).
- Greater variation of system configurations (RTTs, bottleneck link rates etc.)
Formalities
The deadline for handing in your assignment is: Monday, February 19th at (15:00:00.00).
Deliver your report (as PDF) at https://devilry.ifi.uio.no/.
The groups should also prepare a poster (2 x A3 pages) and a quick talk (max 5 minutes without slides) where you pitch your poster for the class on February 22nd. Name the poster with your group name, and e-mail the poster by email to inf5072@ifi.uio.no no later than noon (12:00) on February 21st. We will then print the poster for you.
For questions and course related chatter, we have created a Slack space:
https://mpglab.slack.com/messages/inf5072/
There will be a prize for best poster/presentation (awarded by an independent panel and independent of the grade).
Please check the Amazon VM & Transport Protocol FAQ page for updates and FAQ.
For questions please contact: inf5072@ifi.uio.no
Resources: