I arrived and spent the morning with the rollout team. It is their job to roll out software to the hundreds of store's machines. They use software called EDM (I can't find the webpage) to handle machine configuration and software rollout. Most of the patches they roll out are for the not-great ARMS database frontend (java) that runs PnP. They have a class file that they replace for patches. Their department does nothing but software roll-out (relating to the update process), not even testing, there is a seperate department for that. They also do tech support for a bunch of people, which they seem, understandably, bitter about. This is different to how I imagined things to be. I never thought there would be a department whose sole purpose was to deploy software. Luckily their organisational structure fits quite nicely into my architecture and only some conceptual changes in my mind need to be made, including finding out about more organisational structures and factoring in company politics e.g. department x won't work with department y on project z because they feel it is their job only.

Below is a diagram of which parts of the system are run by which department:

The type II sysadmins seems to be the normal sysadmin types. The policy enforcement will be done by the sysadmins in consultation with management. The 'relevant developer' will most likely by the type II sysadmin's but will occasionally be developers, e.g. for ARMS patches. "Change Control" isn't a department but rather and organisationa wide practice. I heard tale of a network admin group but I will need to find out more about that. The systems database is used by both the type I and II sysadmins, rollout team and probally a few others and is hence a "shared resource". The rest are pretty obvious.

I then had lunch with Pieter at the PnP cafetaria, it was cabbage. After that we had some great coffee at a cafe accross the road.

When I got back I spent some time with one group of sysadmins (we'll call them type I sysadmins). They are in charge of BMC patrol which has the potential to run the entire organisation but is under used so as not to step on other departments toes. It monitors the health of the store's machines using an agent based architecture, making sure that disks, memory, critical processes etc. are all in check. There a bunch of ways BMC can provide notifications of failures like SMS, e-mail, pagers etc. the type I sysadmins have set it up to log an error at the help desk directly, the help-desk can then send it to the type II sysadmins if they can't fix it. It also monitors the health of the database, these notifications are sent straight to the DBA's to fix. If necessary the DBA's can send it to the Data Admin's who look after the actual data in the database. The type I sysadmins recently purchased a distribution server plugin for BMC which looks like it can do a lot of what EDM can do. BMC actually looks like it can do everything EDM does. That's just my speculation.

Change control gets its own section. It is done everywhere and for everything. A plugin for the tiscali helpdesk (can't find the link) is used. The program to interface with it is installed on every machine. It also doubles as a bug-tracker and job scheduler as employees are notified of what jobs they need to do through it. Any changes that are made, however minor are documented and, if necessary, escalated to upper-management for a decision. Their change control is excellent it seems.

The security of the place was completely modeled on what Bill Cheswick, way back during the Morris worm in this paper, described as "a sort of crunchy shell around a soft, chewy center", i.e. using a hardcore firewall while having no or very little internal protection, perimeter security (thanks joat). There were passwords on white boards , faily simple passwords that were repeated and freely shared, unpatched machines all over the place etc. According to people like Barry they are one of the more secure organisations he has seen.

All in all an excellent day where I learnt a lot.