Saturday 18 December 2010

Perforce Metadata Replication

Replication is the duplication of server metadata from one Perforce Server (the master server) to another Perforce Server (the replica server). The basics of replication are covered in Chapter 10 of the Perforce System Administrator's Guide. This article provides answers to common questions that come up when setting up and implementing Perforce metadata replication.

With the release of Perforce Server 2009.2, the p4 replicate utility is the supported method of Perforce metadata replication. Information on system requirements and limits and restrictions when using p4 replicate can be found in the Perforce System Administrator's Guide.

The functionality provided by p4 replicate was first introduced in beta (prior to the release of Perforce Server 2009.2) as a standalone utility called p4jtail. p4jtail is now available as open source in the Public Depot as an example of a client program that uses the Perforce C++ API to do replication. It also serves as a starting point for those interested in extending or customizing replication behavior in some way not supported by p4 replicate.

Prior to p4 replicate and p4jtail, the common utility used for replication was p4jrep, a user-contributed Perforce journal replication utility available from the Public Depot. Both p4jrep and p4jtail remain unsupported but customers can obtain assistance and advice on usage by contacting Perforce Technical Support.

What is p4 replicate?
Where do I run the p4 replicate command?
Does p4 replicate work against archive file content?
Can I replicate from one master server to multiple replica servers?
Can I replicate bidirectionally?
Can I checkpoint the master server?
Can I checkpoint the replica server?
How does p4 replicate extract journal records from the master server?
How does p4 replicate locate journal files on the master server?
How does p4 replicate track what journal records have been processed?
Can I filter what gets replicated from the master server?
What is the upgrade process when doing replication?

Q. What is p4 replicate?

A. p4 replicate is a client command that polls a primary Perforce Server (the master) for new journal records, then provides these records to a subprocess that restores the records into the database of a Secondary Perforce Server (the replica).

Q Where do I run the p4 replicate command?

A. You run p4 replicate on the machine that will host the replica Perforce Server. Because p4 replicate is a client command, you can run it from any client machine that can connect to the master server.

Q. Does p4 replicate work against archive file content?

A. No. p4 replicate only replicates Perforce Server metadata, that is the information contained in the db.* files. Depending on the requirements for your replica server, you can optionally configure the replica server to read the master server's versioned files, or use third-party or native OS functionality to mirror the versioned file content to the replica server.

Q. Can I replicate from one master server to multiple replica servers?

A. Yes, multiple instances of p4 replicate can be run against the same master server.

Q. Can I replicate bidirectionally?

A. No, replication is unidirectional.

Q. Can I checkpoint the master server?

A. Yes, p4 replicate continues to run during journal rotation on the master server, except when p4 replicate is started using the -x option. If p4 replicate is started with the -x option, replication will terminate when journal rotation is detected on the master server.

Can I checkpoint the replica server?

A. Yes. If using the replica server for offline checkpoints, checkpoint using the p4d -jd option. See the Offline Checkpoints Knowledge Base article for further details.

Q. How does p4 replicate extract journal records from the master server?

A. Behind the scenes, p4 replicate makes use of the p4 export command to extract journal records from the master server. In addition to providing journal records, p4 export adds fields to its output to indicate points of transactional consistency and the end of the journal file.

Q. How does p4 replicate locate journal files on the master server?

A. p4 replicate locates the live journal file on the master server based on the setting of P4JOURNAL or the -J option passed to p4d on startup. For rotated journals, p4 replicate looks by default in the P4ROOT directory on the master server and uses the default naming convention for rotated journals (journal.nnn). If using the prefix option when checkpointing or rotating the journal on the master server (prefix in this example is /data/backup/p4):

p4d -jc /data/backup/p4 ORp4 admin checkpoint /data/backup/p4p4d -jj /data/backup/p4 ORp4 admin journal /data/backup/p4

the name and possibly the location of rotated journals will not follow the default convention. In this case, the -J flag to p4 replicate must be used, specifying the same prefix used in the checkpoint and journal rotation commands on the master server:

p4 replicate -J /data/backup/p4 ...

Q. How does p4 replicate track what journal records have been processed?

A. p4 replicate uses a statefile to store a journal position token consisting of a journal number and position (byte) offset indicating what journal records from the master have been processed.

Q. Can I filter what gets replicated from the master server?

A. Yes. Filters can be used with p4 replicate to customize the journal records that are replicated. The command that p4 replicate runs on the replica to replay journal records could be, for example, a script that first applies some filter to the journal data stream and then passes the data to the command used to replay the journal records into the replica.

Warning: If you are using replicate to perform offline checkpoints, then do not filter the journal stream. By definition, filters remove content; therefore, a filtered journal replication will not provide sufficient data integrity for your checkpoint and backup process. The primary use of journal stream filtering is to support replication to a read-only Perforce Server used for build or reporting purposes. Depending on the specific use case, it might be desirable to filter out db.have or other db table entries.

Q. What is the upgrade process when doing replication?

A.  For detailed upgrade procedures, see the Upgrading replica servers section of the System Administrator's Guide.


View the original article here

Installer problems on Vista

Perforce Installer versions 2005.1 and earlier do not run by default on Windows Vista because they do not recognize the OS version. Running these older installers will generate the following error message:

Perforce is only supported on Windows NT/2000/XP/2003 and Windows 98/ME.Windows 95 and Windows 98 Retail Edition are not supported.This installation will not continue

To work around this installation problem, right-click on the Installer executable (perforce.exe) in Explorer, select Properties, and go to the "Compatibility" tab. Set the installer to run in XP compatibility mode with full administrator privileges.

An alternative is to simply use a later version of Perforce and the Perforce installer. As of the 2005.2 version of Perforce, there is no problem running the Perforce Installer on Windows Vista.


View the original article here

Using the server flag -Cn

Please note that the use of -C1 is documented, however the use of other -Cn options are still undocumented, and are subject to change or removal without prior notice. Consider using the p4migrate utility instead to translate case. If you choose to use the -Cn flag, you must ALWAYS use this flag.  This is necessary both when starting your Perforce Server and when running checkpointing or upgrade operations.  If you do not do so, you risk corrupting your Perforce database and/or your checkpoint and journal.

What does the -Cn flag do?

When do I use it?

The -Cn flag instructs the Perforce Server to override the operating system's case handling behavior for the Perforce database. The flag was introduced in release 2004.2, and is a server side flag. From the release notes:

Forces the server to operate in case-sensitive (n=0) on Windows orcase-insensitive (n=1) mode on Unix. -C2 is an experimental 'hybrid'order: case folding but uniqueness preserving.

You use the -Cn flag when:

You want to change the way the Perforce Server handles the case, overriding the operating system's default case-handling.
You are migrating your Perforce Server to an operating system with a different case-handling method, but you wish to retain the existing method for case-handling.

You do NOT use the -Cn flag when:

You are not intending to override the operating system case-handling.

If you do not intend to override case-handling, you checkpoint the database, resolve any case conflicts and inconsistencies, then restore from this case-consistent checkpoint. This process is explained in more detail in Article #75: Cross-Platform Server Migration.

You MUST NOT use -C0 or -C2 when:

running your Perforce Server on a Windows platform, and one file name or path differs from another only in case. This will corrupt the corresponding archive files.

When using either -C0 or -C2, the Perforce Server makes a distinction between "//depot/file.txt" and "//depot/FILE.TXT", and tells the operating system to write "file.txt,v" or "FILE.TXT,v" accordingly. Although Windows is capable of preserving the case used, it is unaware of the difference between these two names. Therefore, "file.txt,v" overwrites or is appended to "FILE.TXT,v", and so too for the reverse, causing loss or corruption of revision content.

If you have not created a database, you can simply start the Perforce Server using the appropriate -Cn flag.

For an existing database, call or email Perforce Technical Support at support@perforce.com because this is not a trivial task. You will need to take a checkpoint of the database, then restore it including the appropriate -Cn flag in the "restore" command. This restore ensures that the order of the keys is appropriate for the case-handling. In addition, you will need to correct case differences using the p4migrate script, and change versioned file line endings.

Important: The -Cn flag can only be used against new Perforce databases. Using the -Cn flag against a Perforce Server that is already in use instantly renders the keys out of order. Not only does it make the data largely inaccessible, it could easily cause the btree code enough confusion to corrupt the tree.

Notes:

The 2007.2 release of the Perforce Server checks the case order of the database on startup. If there is a mismatch, an error is reported in the log:

Perforce server error:Database open error on db.counters!BTree Case Order Mismatch
As of 2010.1 -C1 is now documented in p4d -h $p4d -h$ -C1 Force server to be case insensitive

For a working example of using the -Cn flag see Article 516: Moving from Windows to Linux, Retaining Case-handling.

3 users have rated this article 3.3 out of 5

View the original article here

How to Monitor a Swamped Perforce Server

How to monitor a swamped Perforce server not responding to p4 monitor show commands. In most circumstances, you can run the p4 monitor command to identify the client processes currently running on your Perforce Server. However, it is possible that a long-running Perforce command can block other Perforce processes, including the p4 monitor request.

It is possible to prevent the blocking of p4 monitor on a Perforce Server running on UNIX or MacOS (or other operating systems that natively support symbolic links). To do so, you must create a new Perforce Server instance that shares the db.monitor table by means of a symlink.

Below is an example of configuring a new "monitor server". Note the new monitor server runs in a different P4ROOT (/perforce/monitor) location and on a different port (1999) than the production server. # Create a new monitor server P4ROOT directory #mkdir /perforce/monitor# Link the production server monitor table into the new P4ROOT directory#ln -s /perforce/production/db.monitor /perforce/monitor/db.monitor# Start up the new "monitor server" (with journaling off)#p4d -r /perforce/monitor -J off -p 1999 -d# Remember to enable monitoring by setting the monitor counter to 1 on the new server#p4 -p 1999 counter -f monitor 1# Stop and restart the monitor server for the new monitor setting to take effect#p4 -p 1999 admin stopp4d -r /perforce/monitor -J off -p 1999 -dOnce the monitor server instance is set up (as above), you can then monitor your production server by running the p4 monitor command on the new monitor server. For example: p4 -p 1999 monitor show

View the original article here

Changing File Case

Change the case of a file on a Perforce Server.

The solution to changing the case of files depends on the native case-handling behavior of your platform. For platforms with case-sensitive file systems (for example, Linux), the case change operation is a straightforward rename. On Windows, additional steps have to be taken to account for the fact that the NTFS file system largely ignores case differences; it is case-aware, but case-insensitive.

Because Windows uses a case-insensitive file system, references to files that differ only by case are ignored. In order to get the Perforce Server to change the case of a file on a Windows Server, the file must first be deleted and then re-added to Perforce using the correct case. Additionally, the internal Perforce "have list" references to the old case name must be eliminated by removing the file from case-insensitive workspaces and then re-syncing the file using the correct case.

Windows Example

In this example, file "foo" has three revisions and you want to change the file name to use all upper-case, from "foo" to "FOO".

Delete the existing file p4 delete foop4 submitRecreate the file in your local file system with the correct case. Then, use the p4 sync command to re-create the file in your workspace using the correct case. p4 sync FOO#3Remove the internal "have list" reference. The p4 flush command (sync -k, in recent versions) removes the have list reference without removing the file from your workspace. p4 flush FOO#noneRe-add the file in the correct case. As with the sync command, the p4 add command uses the case of the file argument provided at the command line, so it is important to provide the correct case. p4 add FOOp4 submit

Notes

To obtain the newly case-changed file, all users who use workspaces on case-insenstive file systems (such as Windows NTFS or MacOS X HFS) must remove the old file from their workspace and then re-sync the renamed file with the correct case to replace it in their workspace.

For the example above, this means:

p4 sync foo#none
p4 sync FOO#head

On Unix, simply integrate the file and delete the original file.

Unix example

Use p4 integrate to change the old name to the new name. $ p4 integrate kiwi.txt Kiwi.txt//depot/Kiwi.txt#1 - branch/sync from //depot/kiwi.txt#1,#9Delete the original name. $ p4 delete kiwi.txt//depot/kiwi.txt#9 - opened for deleteSubmit the changes. $ p4 submitChange 12681 created with 2 open file(s).Submitting change 12681.Locking 2 files ...branch //depot/Kiwi.txt#1delete //depot/kiwi.txt#10Change 12681 submitted.

Verify the results

Use the p4 files command to check that the Perforce metadata is in the proper case.

$ p4 files Kiwi.txt//depot/Kiwi.txt#1 - branch change 12681 (text)

Note that "Kiwi.txt" is now in the proper case.

11 users have rated this article 2.7 out of 5

View the original article here

How do I check in a different branch than I check out from in perforce.

According to the Perforce Blog, the p4 move -f command will move your edit from one branch to the other. From the blog

This can be a real lifesaver if (as I did a few weeks ago) you start working on a fix in your mainline, and then decide that it needs to go into your release branch instead. Previously you would have needed to save a copy of your work, revert it, open the files in the release branch, and copy your edits back in manually — now you just “p4 move -f main/… rel/…” and all your edits go where you need them.

This is a new feature added in version 2010.1 (this year I believe) so that's probably why tech support didn't know about it.


View the original article here

Friday 17 December 2010

Perforce / Java Developer - Berkshire - Software and Build Engineer

Perforce / Java Developer - Berkshire - Software and Build Engineer

Huxley Associates are working with a market leading software house in Berkshire to recruit a Java Software Engineer with Systems Build / Build Engineer skills.

Your time will be split across the two different actitivities of software engineering and the software build process. This is going to be a great time to join as they are rapidly expanding based on the growth in their market and contracts they have won with an enviable list of global organisations.

Their software products are written in core Java with an Oracle backend and their build process is through Perforce source control systems (previously used Subversion). They expect you will have excellent knowledge of shell scripts on Linux, Ant, Oracle setup and admin and Tomcat (other apps servers also considered).

If this varied role would suit your skills and experience please apply now for immediate consideration.


View the original article here