Extreme Computing

First practical assignment

The goal of this assignment is to write a 3000-word essay (roughly, 6-7 pages of text) on the infrastructure and uses of cloud computing. More specifically, you are given a description of a computing problem. You have to analyse this decription further, identify how it relates to the issues touched upon during the lectures, and then propose a computing infrastucture to address the problem at hand.

The proposed infrastructure can be in the form of an architectural diagram, or a simple bulleted list of points. There are no requirements set in stone, choose whichever form you think makes more sense for your presentation style. The only requirement is that it addresses the presented use-cases.

Also, this being an essay, means that there is no right or wrong answer. Of course, some answers make more sense than others (for instance, saying "I would buy 10,000 monkeys to write random code for me" is an interesting experiment but does not make too much sense) but the basic rule of thumb is that if you can argue about what you have written, in the sense of exhibiting a clear way of thinking and providing supporting arguments, then the result is perfectly acceptable.

It may well be the case that you disagree with the feasibility of a solution to the presented problem. That is also perfectly acceptable, provided that you present your reasons why you think that a solution is infeasible.

Context

Onefile is a new startup company that is active in the area of cloud-side storage. What they provide is effectively a massively distributed file system for all their clients to use, but with a twist: for any single file resource, there is only one logical copy of that resource. Let us now see what they mean.

What Onefile have done, is to analyse the use of large file systems, as these manifest in various organisations and institutes. They were taken aback by one simple observation: there are multiple copies of the same resource. For instance, one of the organisations they analysed was a Computer Science department of a well-known University. There, they found out that the most common use case was for all students to save a copy of the lectures of their courses locally, below their home directory. So, the 100 students taking the same course had 100 copies of the lecture notes -- each identical to the other. This seemed to the Onefile analysts as a huge waste of space and resources. It was also what gave them the idea for starting up Onefile.

The intuition behind Onefile is that if a resource is the same across users of a file system (or indeed across multiple locations in the file system) it should only be maintained once as a physical entity. The rest are logical aliases for the resource. So, in the previous example, if the same physical resource corresponds to a file named "lecture-notes-exc.pdf", which various students have downloaded and renamed to their local directories, then: (i) there should be only one physical copy, and (ii) the rest should be logical links to that physical copy.

Onefile have the following rough set of specifications/requirements for the service they aim to provide:

As noted above, physical files appear once, but they may have multiple logical aliases spanning both directories of the file system, and users.
As soon as there is a need (for instance, due to one user editing a file) the file is replicated and dettached from its old physical resource.
Even though files are unique, metadata is not. So if two users refer to the same physical file, but using different names and with different privileges, they should see the two different logical versions of the file (so security is a key consideration).
To cope with fault-tolerance, and even though file resources are unique, there should be a good degree of redundancy to allow constant access and also improve performance.
The system is not only an online one, but it should be possible for users to access their files offline as well. So if a user loses connectivity, they should be able to work on their files transparently.
In light of the above, the file system should have a synchronisation capability, so when a user, or a portion of the file system comes back up online after a partition, then the contents are synchronised with the rest of the file system.
The file system should be operating system agnostic. In other words, it should present the same view regardless of the operating system each user actually uses (think of accessing your files through Dropbox -- they are the same regardless of your host operating system).
Onefile aim to start small, so they plan to deploy their system to private cloud infrastructures. However, the design should be extensible, so if there is any wider adoption and people start massively using their infrastructure, the design should be capable of being applied to largely distributed file systems (think of the single namespace of AFS).

Being impressed with you having taken Extreme Computing and your excellent mark in the course, they have decided to put you in charge of infrastructure. What they want you to do is come up with a proposal for the architecture and technology of the underlying system they will build and use. Specifically, the investors have given the company an infrastructure budget to be spent over the next two years in any way the company sees fit -- at which point the investors in the startup will evaluate whether Onefile will receive further funding, or they will pull the plug. You responsibility in Onefile is to make the best use of this budget -- in addition to providing the overarching system design and identifying where the technical effort should be spent.

At the end of the day, you need to design their file system and decide which of their requirements are realistic, which are infeasible, which can be built in the given period of time, etc. Questions to think about:

What sort of technology does Onefile need? What capabilities does that technology have and what parts of the envisioned functionality can it provide?
Are some of the goals unreachable? If so, which goals should be dropped from the specification and why?
How far can they go in building their envisioned system? How should they identify file resources? How should they detach resources on change? What sort of data structures do they need to separate data from metadata?
Fault tolerance implies replication. But replication needs to be at the level of a physical file. So changes to a replica needs to propagate across replicas before they can be assumed as having taken place. What sort of consistency models and semantics can they achieve, especially as they scale the size of the system?
How about security? Is it feasible to have different sets of metadata for a single physical resource? How should they go about implementing it?
How about platform independence? Different metadata will have different manifestations on different operating systems. How should they achieve this?
In terms of deployment to larger user bases, how realistic is it to touch this with a single overarching design? How can they prove extensibility at this point?

Structure

Below is a potential (and rough) structure for your essay. You do not need to follow it, but it might be helpful in terms of organising your thoughts:

Introduction
Design considerations (what are the salient aspects of the problem, and what technology currently exists to support them?)
Alternatives (including advantages and disadvantages of each)
Proposed solution (including arguments of why you think it best fits the problem at hand)
Conclusion

Marking guidelines

There is a total of 100 marks available. The marks will be awarded as follows:

10 marks for writing quality (so please proofread your essay before submission);
40 marks for presenting the use cases and commenting on their feasibility. You should identify any common patterns that you have used to provide your proposal; note that you need to focus not only on what is already possible, but also on what needs to be done and what should be dropped as a goal;
20 marks for the proposed solution and how well it aligns to the presented arguments;
20 marks for your arguments and well they are supported by existing work;
10 marks discrectionary (e.g., general style of writing, exhibiting off-the-beaten-track thinking, etc.).

Submission

The submission is electronic only and the deadline is Wednesday, 7 November, 4:00 pm.

You will only need to submit a single PDF file of your essay (note: no word documents, or open-document format files). Use the submit program to make the submission. Name your file exc-essay.pdf; you can submit it from the command line as:

submit exc 1 exc-essay.pdf

That is all. For any questions, contact me.

Good luck.
--stratis

Home : Teaching : Courses : Exc