The purpose of this document is to describe the software design of the Vasudeva Service Application, Generation 3. This document uses the terminology established in the System Requirement Specifications.

ZEO/Zope/Storage Configuration

The configuration described here applies specifically to the instance of the Vasudeva Service Application on the Vasudeva Servers.

Production ZEO server

The production ZEO server runs in the directory /opt/zope/vsa_3/zeo/. It manages several filestorages (ZODBs) under /opt/zope/vsa_3/var. Some of these are used by Core Developers, Site Administrators, Editors, and Designers only, while others are used by all users. This is described below.

Development Zope Instance

A development Zope instance for Core Developers currently runs in the directory
/opt/zope/vsa_3/dev_client/. This Zope instance is used for development of new content-type components and diagnostic components. Through a process involving the development Zope instance, components are tested in a staging area and later pushed to a production area, where they are visible to all users. This work flow is described later.

The development Zope instance mounts a filestorage in its root, and mounts zeoclients in staging and production Folders. One of the production zeoclients is a development fishbowl. The root filestorage contains developer sandboxes and User Folders; thus, the frequent writes in developer sandboxes do not tax the production ZEO server.

In the future, it is expected that the development Zope instance will move to another machine, to free up machine resources for Members and Anonymous Users. At that point, only activity under the zeoclient mount points would impact the physical server on which the production ZEO server runs. In particular, work in the developer sandboxes would have no impact on that server.

Public Zope Instance(s)

A public Zope instance for Site Administrators, Editors, Members, and Anonymous Users runs in the directory /opt/zope/vsa_3/pub_client/. In the future, this instance will be replicated on at least one additional physical server.

The public Zope instance mounts a filestorage in its root, a zeoclient for persistent session storage (so sessions are not lost on restarting the Zope instance), and several zeoclients in a production Folder in Zope.

Note that Members may generate frequent writes against the production ZEO server. The public Zope instance manages all Zope user accounts and Plone Member Folders. The Zope acl_users Folder for some of these accounts will be managed outside of Plone Sites so that several Plone Sites may share memberships, while maintaining distinct Member home pages.

Administrative Zope Instance(s)

An administrative Zope instance runs in /opt/zope/vsa_3/adm_client/. The Zope Administrator uses this instance to create new sites and to diagnose problems with components of the Vasudeva Service Application. The administrative Zope instance has a flexible storage configuration. It is typically shut down when not in use.

Storage

Public

/opt/zope/vsa_3/pub_client/var/Data.fs

This is mounted as a filestorage at the root of the public Zope instance(s).

/opt/zope/vsa_3/var/pub/session/Data.fs

This is mounted as a zeoclient by the public Zope instance(s). This provides persistent storage for all Member and Anonymous User sessions.

Admin

/opt/zope/vsa_3/adm_client/var/Data.fs

This is mounted as a filestorage in some configurations of the administrative Zope instance.

Development

/opt/zope/vsa_3/dev_client/var/Data.fs

This is mounted as a filestorage at the root of the development Zope instance. It holds developer sandboxes and developer User Folders.

Staging

/opt/zope/vsa_3/var/stage/srichinmoyraces.org/Data.fs

/opt/zope/vsa_3/var/stage/srichinmoycentre.org/Data.fs

/opt/zope/vsa_3/var/stage/srichinmoy.org/Data.fs

/opt/zope/vsa_3/var/stage/srichinmoylibrary.com/Data.fs

/opt/zope/vsa_3/var/stage/vs_sites/Data.fs

These are staging ("testing") areas for the respective domains. Each domain may comprise several locales. In the development Zope instance, work is pushed here from developer sandboxes and tested. The development Zope mounts these as zeoclients under /home.

The production ZEO server will continue to serve the staging area after the development Zope moves off-server, so that the staging area can mimic the production area as closely as possible.

The filestorage under vs_sites contains a group of sites, such as radiosrichinmoy.org and vasudevaservice.com, the development fishbowl.

Production

/opt/zope/vsa_3/var/pub/srichinmoyraces.org/Data.fs

/opt/zope/vsa_3/var/pub/srichinmoycentre.org/Data.fs

/opt/zope/vsa_3/var/pub/srichinmoy.org/Data.fs

/opt/zope/vsa_3/var/pub/srichinmoylibrary.com/Data.fs

/opt/zope/vsa_3/var/pub/vs_sites/Data.fs

These are the production ("live") areas for the respective domains. Each domain may comprise several locales. In the development Zope instance, work that has been tested is pushed into these areas.

In the development Zope instance, these are mounted as zeoclients under /pub. In the public Zope instance(s), these are mounted as zeoclients under /home.

The filestorage under vs_sites contains a group of sites, such as radiosrichinmoy.org and vasudevaservice.com, the development fishbowl.

Example

The following summarizes the intended use of the storages described above, using srichinmoyraces.org as an example.

The public Zope instance(s) ZMI has the following:

a. /home/srichinmoyraces.org (production area)

Members will have write access to this Zope, and will interact with the Plone Site object srichinmoyraces.org through the Plone administrative interface.

The development Zope instance ZMI has the following:

a. /pub/srichinmoyraces.org (production area)

b. /home/srichinmoyraces.org (testing area)

c. /dev/medur/srichinmoyraces.org (sandbox)

d. /dev/tejaswi/srichinmoyraces.org (sandbox)

The development Zope instance and public Zope instance(s) should have the same access rules in their respective Zope roots.

The two mounts indicated by "a." refer to the same physical filestorage on the file system, even though their mount points are named differently.

Procedures

Sandboxes

Each Core Developer, Designer, and Site Administrator has a dedicated sandbox area, as indicated by the naming convention above.

Two users, such as a Core Developer and a Designer, can pull from each other's sandboxes if they desire. A user should not push into another user's sandbox without permission, for obvious reasons. This action is not prohibited by the system, but can be managed through standard Zope security by the user, who owns his own sandbox.

Staging

The following procedure pertains to work done on a single site through the development Zope instance. The Site Administrator is responsible for ensuring that the process is followed.

When work on a site, such as a portal slot, is a candidate for prime time, a staging area for the site is prepared. In order to duplicate the production site, the filestorage containing the staging site is replaced by the filestorage containing the production site. This requires a restart of the development Zope instance.

The site is represented as a mounted Plone Site object in the /home Folder. If this Plone Site object (=mount point) is deleted or replaced, the mount is broken and the actual Plone Site object inside the filestorage managed by the associated zeoclient is untouched. If not remounted but replaced, the new object is stored in the root filestorage of the development Zope instance, not in the zeoclient for that staging site. Further, replacing the large Plone Site object by copy-and-paste will be very expensive; it will take a long time and will bloat the root filestorage with undoable transactions. In order to keep the testing activity in the zeoclient (to most closely duplicate production) do not delete or replace the mounted Plone Site object. If it is accidentally deleted, just remount it.

Testing is carried out by a user who did not create the new objects. The Site Administrator is responsible for ensuring that adequate testing is done.

Special procedures must be followed for objects that can not simply be copied and pasted through the ZMI. These include the following.

• Site-specific External Methods

Methodology: External methods follow the Zope instance. Python code is managed in cvs on the file system. Tests are performed against the head revision.

• Products, including custom Archetypes

Methodology: Products follow the Zope instance. Python and configuration code for custom archetypes is managed in cvs on the file system. Tests are performed against the head revision.

If changes to a staging site cause the development Zope instance to spin or crash, that site will be unmounted and taken off-line until it is corrected.

Production Deployment

The following procedure pertains to work done on a single site through the development Zope instance. The Site Administrator is responsible for ensuring that the process is followed.

After successful testing, all new or changed objects are identified and copied to the production server. The Site Administrator is responsible for the process of copying objects to the production Folder.

The site is represented as a mounted Plone Site object in the /pub Folder in the delelopment Zope instance. If this Plone Site object (=mount point) is deleted or replaced, the mount is broken and the actual Plone Site object inside the filestorage managed by the associated zeoclient is untouched. This means that the public Zope instance does not see any change. If not remounted but replaced, the new object is stored in the root filestorage of the development Zope instance, not in the zeoclient for that staging site. Again, this means that the public Zope instance will not see any changes, particularly the changes. When pushing changes to production, do not delete or replace the mounted Plone Site object in the /pub Folder. If it is accidentally deleted, just remount it.

Special procedures must be followed for objects that can not simply be copied and pasted through the ZMI. These include the following.

• Site-specific External Methods

Methodology: External methods follow the Zope instance. Python code is managed in cvs on the file system. Tests are performed against the head revision.

• Products, including custom Archetypes

Methodology: Products follow the Zope instance. Python and configuration code for custom archetypes is managed in cvs on the file system. Tests are performed against the head revision.

If changes to a production site cause the public Zope instance to spin or crash, that site will be unmounted and taken off-line until it is corrected.

Maintenance and Diagnostic Scripts

The Vasudeva Service Application includes scripts to handle the following tasks.

• ZEO daemon restart on system startup

• Zope instance daemons restart on system startup

• Zope instance scheduled restart

• Demonstration that session is preserved across public Zope instance restart.

• filestorage scheduled packing

• Log rotation

Application Installation

Following are the components of the Vasudeva Service Application, including both third party products and in-house contributions.

python 2.3.3

Install this in /usr/local/lib/python2.3.3. Install as root.

Zope 2.7.0

Installed in /usr/local/zope-2.7.0. Install as zope.

Zope Products

Core products sit in the Zope installation area. Other products travel with the Zope instances, and go in the Products directory in $INSTANCE_HOME.

Archetypes 1.2.4 on python 2.3 requires that you install the generator and validation python packages. They go in the site-packages directory of the python installation.

Plone 2.0 travels with the Zope instance.

External Methods

External Methods travel with the Zope instance and are held in cvs on the file system.

Miscellaneous Topics to Incorporate Later

Using / Not Using zdctl

http://zope.org/Wikis/ZODB/HowtoUseZdctl/zdctl.html

This may be fine for testing, but a bad idea for running a ZEO server in a production environment. In production, you want the ZEO server to be run as a daemon process, you want the log output to go to a file, you want the ZEO server to be started when the system is rebooted, and (usually) you want the ZEO server to be automatically restarted when it crashes. You should also have a log rotation policy in place so that your disk doesn't fill up with log messages.

The zdctl/zdrun combo can take care of running a server as a daemon process and restarting it when it crashes. It can also be used to start it when the system is rebooted. Sending log output to a file is done by adjusting the ZEO server configuration. There are many fine existing tools to rotate log files, so we don't provide this functionality; zdctl has a command to send the server process a SIGUSR2 signal to tell it to reopen its log file after log rotation has taken place (the ZEO server has a signal handler that catches SIGUSR2 for this purpose).

Daemonizing ZEO

http://zope.org/Wikis/ZODB/HowtoRunAZEOServer/howto.html

Running the ZEO server as a daemon

Persistent Session

Session Storage setup (Evan Simpson)

Description:
Sessions are great, and the default Zope configuration (keeping Sessions in a TemporaryFolder) works well as long as [1] you don't restart Zope and [2] a user's requests are always served from the same Zope instance.

If you have multiple Zope instances tied together with ZEO, or you just don't want to wipe out all your site's sessions when you restart Zope, you need persistent sessions. With the integration of Shane's DBTab into Zope 2.7, and the nifty new ZConfig, it's not that hard to set up.

The following assumes that you've used the standard (as of 2.7) 'configure/make/make install', 'bin/mkzopeinstance', and 'bin/mkzeoinstance' process to make an Instance Home.

---

Source (Text):

o Add to the Instance Home's 'etc/zeo.conf' file these lines::

  <filestorage main>
    path $INSTANCE_HOME/var/Data.fs
  </filestorage>

  <filestorage session>
    path $INSTANCE_HOME/var/Session.fs
  </filestorage>

o Add to the Instance Home's 'etc/zope.conf' file these lines::

  <zodb_db main>
     <zeoclient>
       server $INSTANCE/var/zeo.soc
       storage main
       name main
       client main
       var $INSTANCE/var/
     </zeoclient>
     mount-point /
  </zodb_db>

  <zodb_db session>
     <zeoclient>
       server $INSTANCE/var/zeo.soc
       storage session
       name session
       client session
       var $INSTANCE/var/
     </zeoclient>
     mount-point /session_folder
  </zodb_db>

  <zodb_db temporary>
     <temporarystorage>
       name temp
     </temporarystorage>
     mount-point /temp_folder
     container-class Products.TemporaryFolder.TemporaryContainer
  </zodb_db>

o Note: adjust the 'server' lines to fit your ZEO configuration.  The example assumes that you are using a Unix socket in the same directory as the databases.

o Start ZEO and Zope using 'bin/zeoctl start' then 'bin/zopectl start'.

o Wait until 'log/event.log' exists and shows a successful startup.

o Open the ZMI in a browser.  Add a 'session_folder' Folder to the root.

o Add a Transient Object Container named 'session_data' to '/session_folder'.

o Edit the Transient Object Container Path of '/session_data_manager' to '/session_folder/session_data'.

---

Explanation:
This configuration tells ZEO to create a 'Session.fs' FileStorage in addition to the typical root 'Data.fs', and mount it on top of the Folder '/session_folder'. Since we are overriding the default zodb_db setup, we also need to provide an explicit configuration for the TemporaryStorage that Zope normally mounts on '/temp_folder'.