wissel.net

wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

The 2,147,483,648 NoteId limit

Every Note in a Lotus Notes database has 2 identifiers: a 32Byte Hex Universalid (UNID) and a NoteId (actually there are some more). The UNID is assigned once, never changes (unless you force it), is derived from a timestamp and stays unique across all replicas.
The 128Bit (incidentially the same size as IPv6) are divided in the first 64Bit from the replicaID and the second 64Bit for the individual documents (but you could overwrite that). Normal use gives you 2 64 possible values ( = 1.8 19 = 1,800 quintillion) for documents.
The Noteid on the other hand is unique only to one given database and changes when you create a new replica or run compact -c. For performance reasons (Notes is around for a while) this is a 4 byte Hex number (where only even numbers are used). For backward compatibility in the API that hasn't changed yet.
So you have 2 31 = 2,147,483,648 NoteIds at your disposal. A NoteId is assigned when a Note is saved in a database (could be design or data) and never reused even after a document is deleted and the resulting deleting stub expired and is removed.
When you have a very busy (large) database where many documents are constantly created and deleted you might hit that ceiling, even when the document count and database size seem moderate. The error condition is documented in Technote swg21220384 (the error can also occur in normal operation).
To my best knowledge you can't setup a DDM probe (or any other easy admin tool) to monitor " NoteId exhaustion". A suitable preventive remedy is to schedule regular compact -c for your busy databases.
Be careful with that, since the dose makes the poison and running that task every day will compound the side effects. If your database does need a daily compact, you have a structural problem in your application - come and see me. Typical intervals are weekly or monthly. Smart admins spread them out (so some compact is running every day). The side effect for compact -c on Windows (applied to SAN too) is disk fragmentation. So make sure you take care of that.

Posted by on 08 September 2012 | Comments (3) | categories: Show-N-Tell Thursday

Comments

  1. posted by Nathan T. Freeman on Sunday 09 September 2012 AD:
    "The 128Bit (incidentially the same size as IPv6) are divided in the first 64Bit from the replicaID and the second 64Bit for the individual documents (but you could overwrite that)."

    This is not entirely accurate. The first 64bits are randomly generated within the context of the originating file. The second 64bits are the timedate stamp of the creation of that particular note.

    From what I can tell, these are assigned when the document is created in an NSF. So you have foo.nsf, and you create a document in the UI, or you use an API to create one -- at that time, whether you have saved or not, the document has a UNID. It doesn't get a NoteID until you save it and it's written to the ODS.

    That second 64bits is very important, by the way, because it drives the result of the API's created time for the document. When you forcibly override the UNID, you're invalidating that information, and it becomes important to then make sure you track it yourself.
  2. posted by Dragon Cotterill on Monday 10 September 2012 AD:
    So if we have 2,147,483,648 documents. Lets assume we create one document a second. That would be 35,791,394 minutes.
    Or 596,523 Hours.
    Or 24,855 Days.
    Or nearly 70 years.

    Lets just hope you don't need to update a view index or FT search that database. You'll be waiting just as long for the indexing to finish. Emoticon shocked.gif
  3. posted by Nathan T. Freeman on Monday 10 September 2012 AD:
    @2 I've been testing some code lately that creates around 4000 docs/second, so we could shave that 24855 days down to about 6 days.