Synchronous Languages
Posted by eks in Status Update on September 22nd, 2009
Posted a quick brainstorming page about synchronous languages which may be the seed to the final design. Would solve and put together a few concepts I have been toying with these past few years.
Multi-core, distributed applications, asynchronous services
Wanted to share a few of my latest reads and studies. First is the programming language “Axum” by Microsoft; a good read. It talks about channels which are used to communicate to services using ports with the ability to define states and transitions, define protocols. A quick google search should turn you into an “Axumite”, be careful!
Also came across Reactors, where there are no functions, channels, methods or services. Only data, states and relations. Services are provided by requesting a state change. An interesting read indeed.
How will this all tie in with Unununium? no idea! But if you put together multicore, OpenCL, CUDA, and all the multiprocessing going on nowadays, you can start formulating an idea. Have fun!
Security, Execution Privilege Levels, and the Past
It is surprising how sometime a simple decision can take so much thinking, how it can affect so much the overall design, and how hard it is to settle down on a final choice then say “that’s it”.
The overall role of an operating system is to provide the tools required for a software to execute, in order to simplify the life of a programmer in developping the tools and applications required to perform a specific task. With that in mind, then comes the ability to run multiple simultaneous applications, protection and safeguards between these applications, how these applications will co-exist and have to share access to the same resources.
Looking back at the origin of Unununium, it all started back when V2_OS was identified as seriously lacking proper design and most of its developers agreed to go for a rewrite. The only problem, these developers couldn’t agree on implementation specific ideas and the whole thing pretty much stagnated for some time. Myself and a few folks went on to create our own “4th Axis” way of doing things.
4th Axis, as with V2_OS, was aimed to have as little barrier between the software application and the hardware, providing as little as possible abstraction so the entire system would run as fast as possible. This contrast with common operating systems like Linux, Windows and MAC OS X where applications are severely limited in how they must access the hardware, that they will be pre-empted, they can’t access the memory of another process, etc. Not that these things are bad, they are helping ensure stability in a world where virus writers are king of the hill.
With the introduction of Unununium, 4th Axis lost its kernel; at least in concept. Some ideas were introduced like the ability to reload any component in the system no matter how critical it is; so that a reboot would in theory never be required and no component would have to permanently stay in memory.
So now I’m left deciding where this system is headed, whether I implement paging, multiple execution privilege levels, prevent applications from accessing other processes memory or simply allow the whole thing to simply run with as little hinderance as possible.
As there are many other choices all fighting to make the fastest and most stable secure operating system, and I’m not planning on having Unununium run on all machines on Earth and take over the world, the original ideas of V2_OS, which is the root of all this, should be maintained.
Unununium should be a set of tools and libraries to the software, not a security watchdog. In this essence, Unununium should allow a software application to shut down task switching, or to inspect the entire system memory. Similarly, it should also allow a software to perform I/O operations as desired. After all, what the world need is a good “introduction” operating system which can be used to learn how computer works so that they can then step-up to more complex and security intensive systems.
Unununium should come with a set of tools to do multi-tasking, to debug a software, to trace memory accesses. It should come with libraries to access the network, setup the video card, graphic libraries, etc. Out of all that, it should also allow an application to overtake any of these facets, to replace these components or disable them and even unload them from memory if required. The idea, is for the operating system to be there to help and complement the software.
With this in mind, now I can concentrate on planning how this will all come together..
Memory Management
Posted by eks in Status Update on August 15th, 2009
Finally decided on how memory will be managed. In short, processes will be able to request memory pages of 4KiB to be allocated. The memory pages, as well as the processes, are tied in to a given cpu/core so as to avoid false cache sharing between threads. Some sub-routines will also be available to sub-allocate these memory pages into any allocation size; although the operating system will only keep track of the pages and not the sub-allocations.
The allocated memory pages can then be shared with other processes for fast and efficient data transfer; when doing so the process sharing the memory page will receive a secret code that can be used to cancel the sharing and regain full control. When shared, memory pages are in read-only mode for all users including the original process which shared the page.
Brainstorming
Posted by eks in Status Update on August 9th, 2009
Started a few brainstorming pages about how I could see the system implemented in the future. I hope to continue to develop those sections and eventually write up a set of specification documents. Feel free to comment on the design.
A difference in return
I have recently been toying with various methods of returning from a procedure call; how to indicate an error condition or generate some kind of exception.
Here’s the scoop; the standard methods used in C and other languages are far from being the most efficient, even the ones we used in past Unununium implementations could be optimized further. Here’s how to do it.
First it is important to understand how a cpu is processing instructions. Even before an instruction is executed, a prefetch queue is filled with the next few instructions to execute. The CPU constantly analyzes if instructions can be piped in parallel and if it can pre-determine the result of a conditional jump prior to reaching that point. When the CPU takes a wrong guess, this prefetch queue has to be flushed (emptied) causing some delays in execution.
A standard call implementation such as those found in C and typical languages, uses a return value to indicate if an error occured; such as returning -EOF when performing a fread() to indicate the End Of File has been reached. The program calling the fread procedure (should) always check if the number of bytes read is equal to -EOF or some other error condition; this check takes some cpu cycles to execute and requires a conditional jump which in turn could be wrongly guessed by the CPU and cause a flush of the prefetch queue.
Newer processors and languages allow some flags to be inserted before the conditional jump to indicate what is the most likely outcome to help the processor take the right guess; if something happens and the condition is not matching the typical case the cpu will still have to go thru a prefetch queue flush.
A method exists to alleviate and reduce the number of comparisons required after a function call; not returning the status of the operation at all! While this may sound counter productive, it is actually quite efficient and allows for proper error handling. Here’s a snippet of assembly code to demonstrate a procedure call:
…
jmp short .perform_file_read
dd .invalid_file
dd .end_of_file_reached
.perform_file_read:
call read_file
… use the data, no need to check for errors
.invalid_file:
… do something; abort?
.end_of_file_reached:
… terminate the operation
Note that just before we perform a function call we do a inconditional jump, skipping over an “exception” table. An unconditional jump is easily handled by the CPU and will not affect the prefetch queue in any significant manner. When the call is performed, the ‘read_file’ function is able to calculate the location of the exception table based on the address of the return point that was pushed onto the stack.
In the ‘read_file’ function, something like this can be done:
… check if file handle is valid
jc .is_invalid_file_handle
… check if the end of file has been reached
jc .end_of_file_reached
… read the requested data
retn
.is_invalid_file_handle:
pop edi
jmp [edi-(5 + 8)]
.end_of_file_reached:
pop edi
jmp [edi-(5 + 4)]
Under normal condition, if the ‘file_read’ operation succeeds, it simply return with the data as it normally would, there is no extra cpu cycles used to check if the operation was performed properly and the cost of the entire operation is about the same as with a standard call; with the advantage that the caller still does not need to perform a check!
If an error occured, such as the end of file reached, the CPU will retrieve from the exception table the address where execution should be resumed. The ‘jmp [edi-(5+4)]‘ in the example above is the operation of retrieving the return address from the exception table. When a ‘call’ instruction is performed, the return address immediately located after the call is pushed onto the stack. The encoded form of a near call instruction is 5 bytes long hence the ‘5′ in the code above. By substracting 5 to the return address, we obtain the address of the ‘call’ instruction itself. By substracting another ‘4′ we obtain the address where the last exception handler in the table is located, in this case ‘.end_of_file_reached’.
The advantage to the caller is the code will automatically resume where it needs to without having to do any return value comparison. In a simple malloc() case where the value returned is either NULL (0) or not, a gain of approximately 5% (on the call/return code alone) can be seen using the above described method.
Now, I’d rather call the list of return addresses placed before a call an ‘alternative return points’ table rather than an exception table, as the cost associated is much more minimal, and the table has to be specified at every procedure call whereas an exception table is normally set and shared for a block of instructions.
I created page with the sample bootable image which compares performances of both standard and multipath calls Research: Multipath Calls
Enjoy the increase in performance!
Rules of lobby os development
So, you want to develop a small operating system, then it starts to be popular and suddenly you are overhelmed, running short on time and eventually you close up shop. What happened?
Well, if you are like me what happened is:
- You attempted to support everyone’s machine; at least everyone that wanted to help. The problem is, you can’t possibly write drivers for all the hardware, hell even the companies making the hardware struggle to write proper drivers. What to do? Develop for your machine only, and let those contributor of yours come up with their own drivers, stick to your guns as you have no time to lose supporting other platforms. If you really want everyone to be able to run your OS develop for some virtual machine, like Bochs.
- You probably had the next great idea, but that required a partial or even a complete rewrite; this can be a sign that you either are making great progress or that you completely lacked planning. What you can do is evaluate if NOT doing these major changes would prevent your system from reaching its goals. Develop the system to reach your original goals first and only then consider rewriting it differently. Changing how the system completely behaves or how its designed affect the moral of your followers and contributors. All the work they put in will have to be redone (even if only partially), they will have to learn the new system and they may even disagree with the changes.
- Some fans or contributor comes along and they love your system, and they convince you to extend the original goals to do more; like you know, become the Next Big Thing or their new desktop. Well, Linux has been trying to do that for years and only started recently of getting a bit of hold on the desktop. Don’t try to play the big leagues right away; continue on your path until you have the system completed to your original goals (unless they really and truely don’t make sense anymore). Once your system is sufficiently developped to perform the duties you originally planned, you can sit down and start planning its expansion. Trying to develop the system for too many uses right away will only spread your energy too far and you are going to run out of time. Again, let your contributors contribute and let them come up with the modifications required to perform these extra duties.
- People hear about the system and they want to help you, but they require a little training… and then some more. Don’t try to guide everyone by the hand and teach them how to write for your operating system. Create documentation and when people ask a question, let other followers answer questions and concentrate on planning and developing. If you notice a repeating question, then feel free to update your documentation; you created documentation right?
- You know your goals, but the road to them is a white canvas. This could become problematic unless you have an innate talent for project planning and an awesome memory; but then if you are so good at project planning you probably would have a plan, wouldn’t you? Create a roadmap, you should have documented steps for every 40 hours of labor or so. Keep track at least a loose track of the time you spent on the various phase of each step and add them up; then re-adjust the remaining steps of your roadmap accordingly. By seeing those milestones go by will allow you to keep track of your progress and encourage you to continue. It will also allow your followers to have a better grasp of what’s yet to be completed (and maybe even help along).
Its no secret, writing an operating system is a very complex and long endeavour but you can get started really quickly with very little code. Unless you establish yourself clear goals and a proper roadmap to get there, your chances of success are limited.
The Internet is a big place and you can develop quite a following. A large following without proper planning can bring the project to a grind. If you aren’t a good project planner, then keep the team to a very small crew, potentially yourself only.
Overall, remember that your operating system is a project. As such, it should be managed like a project. Reading books on project and team management may be as important or more important than your ability to write a programming language.
Welcome
Welcome dear visitor,
It is with pleasure that this site is now back up and running although it is with great sadness that I must admit this project hasn’t seen any development in quite a while.
This project has been a host of activities for quite some time, a hub for young developer enthusiasts to collaborate and learn the intricacies of operating system development. For many of us, this was a jumping platform to the commercial world, for others, it was a fun project to hang with friends; for me it was both.
The sources of the project can be located on the CVS, as well as some downloadable images which are still in the download section of our SourceForge’s project page.
I hope to have some time to clean up the source code, generate some kind of neatly organized final release. In the meantime, please feel free to send us your comments!