Forum OpenACS Q&A: RFC: Avoid exposing object_ids in permanent URLs

https://openacs.org/forums/message-view?message_id=75455

I don't want to further clutter that thread, so I'll deal with it separately.

If you expose raw object_id's in permanent URLs, such as the URL of a forum, the URL of a posting, or any other URL which Google will index or users will bookmark, you're setting yourself up for trouble.

Because when at some point you need to migrate your web site to new software, whether it be a new release of OpenACS, a new cool .NET or Java server system, or whatever it is, all of your permanent URLs are going to have to change.

This problem doesn't occur if you stick with OpenACS and run upgrade scripts and everything behaves well. And perhaps this is the situation we want to design for.

But in my mind, object_id is an internal implementation-specific detail, just like the old ".tcl" or ".adp" extenstions were, and thus shouldn't be exposed in the URLs.

Instead, what I'm proposing is what I did with the bug-tracker. Bugs are still ACS Objects, and thus have a bug_id, references acs_objects.object_id.

But they also have a bug_number (could've been named bug_no), which is unique (package_id, bug_number). This is the number that's exposed in public URLs, and this number starts with 1 for every new package, and grows from there.

I think this leads to much nicer URLs and user interface (imagine if instead it was "Bug #830234" even though it was the first bug created in your project.)

It's like in an object oriented system. You wouldn't want to expose the internal object pointer. Instead you expose some reasonable unique identifier (in this case the combination of URL and bug_no).

Makes sense to people?

/Lars

I think there is a lot of value to making moving things around on the site easier and I think having a generic object dispatch could be a great thing (especially in the context of aggregated site views like a new-content page and for searching).

If being able to ensure a range of object_ids are available, we could put a hook in to the bootstrap loader to start at some higher number so that you could find the highest number on the current site, start above that and hence ensure the range is available. Of course if you are merging two sites all bets are off, but I am not sure that's a big deal.

I agree that the ambiguity is a problem.

Another way of fixing that would be to have a bug-tracker instance have a short pretty name, like "OpenACS Bug", or something even shorter, which can be used in referring to bugs, like "OpenACS Bug #123".

This would also be the string that you put in brackets in the subject line of email notifications going out...

OpenACS Bug #123: Here's what we tell you.

When you move a bug, well, you've moved the bug. Conceivably we could keep the old bug row around and clone it, and have bug-tracker redirect to the new, moved copy. The new one would still have a new number, though, as in "dotLRN Bug #73".

/Lars

 
constraint bt_bugs_bug_number_un
  unique (project_id, bug_number)

I am not sure how you do that with postgresql though and if that works in a transaction or not.

Some packages have the notion of what I call "sub instances", e.g., in my initial sketch of Curriculum (just to name something I'm familiar with) an individual instance of it can hold any number of "sequences" (courses) which in turn has "activities" (learning resources)... So, to implement this according to your proposal (which I have nothing against, if it proves valuable), I would have uniqueness on three colums (package_id, sequence_id, activity_no).

Packages designed under the paragigm I described may suffer even more, performance-wise. *If* it is a problem in the first place, that is.

Also, packages will be even harder to design right for beginners.

FWIW,

/Ola

advantage:
document change => key change
key not in current server => could be located in the web, should the key contains that info

disadvantage:
MD5 key is long, 32 bytes
MD5 is not collision free, but no collision known
quite easy for document, post, images but for a bug

note that malte post a similar idea https://openacs.org/forums/message-view?message_id=65365

Subject: Re: [ aolserver-Bugs-588470 ] Memory leak when running Tcl scipts
Bugs item #588470, was opened at 2002-07-30 03:59

I was just looking at a bug email from Source Forge. This one is for AOLserver. Somehow I doubt that AOLserver has had 588469 previous bugs. I do see the desire to have nice small numbers for certain sequences, but first the vendor supplied sequence routines should always be used for that purpose, and they should probably be inside the __new pl routine that creates the object. However, then you are stuck trying to design new ways to return the new 'nice id' so that you can do things like redirect to the right place after the insert. All of the routine ways of doing things in OpenACS are now going to be ad-hoc, and for little or no advantage.

It is useless to consider moving the data in the design of a site. The problem of conserving external links will be the least significant problem. If you move to a different software, more than likely, it will not have the concept of a package either, so the package_id will present the same issue that the object_id presents.

What would be helpful would be an xml or other hierarchical language method of describing the data in your system. This language would aid in the creation of the datamodel, as well as the ability to dump/restore the data.

I think it would still be near impossible to move to another system when you consider the permissioning scheme the package will rely on.

Overall I don't think having big, not quite sequential numbers is a bad idea. It lends a little obfuscation to the workings of a site. It can make it more difficult to hack. For all the benefits you get from not having to deal with new sequences, and getting it wrong, not having to write extra code which is non-standard, I don't see why this is good. For application specific problems like this, maybe create an unique index on some fields in your primary table, I doubt the object_id foreign key is the only thing that makes the row unique. For the bug tracker maybe who submitted the bug and the date is enough. I doubt that one person would submit two bugs at the same time.

I have one table with all the last keys, which can also be initialised to logical start values and an increment value as well as a limit, where the numbering restarts.

This is some overhead but faster than the aggregate function solution and it leaves you in control. As it is in one place its functionality can be changed or improved without changing the logic in the different places of usage.

As it is a logical key (usually the table name) one can use any combination, i.e package-id+subsite-id etc).

This is just a simple approach and in now way the most sophisticated solution. It had to be done quickly and it does it's job since very well.

I insert the code here (out of my environment) just for illustration.

--  DBTYPE=pgs7, generated by zt_gen_sql at Tue Jan 14 16:29:16 GMT 2003
--  for PostgreSQL-7.x, to be executed by psql
--  (C) 1996/99 by Panorgan AG, CH-8820 Wädenswil

--  TABLE taf_idref - TAF ID Referenz Verwaltung

create table taf_idref
(
  i_idref    char(20) not null,    -- ID Referenz Name
  i_last    decimal(10) not null,  -- ID Letzter Wert
  i_start    decimal(10) not null,  -- ID Startwert
  i_incr    decimal(4) not null,  -- ID Inkrement
  i_limit    decimal(10) not null,  -- ID Limite
  primary key ( i_idref))

# ==========================================================
# y_db_getId -- generate uniqe id's for key fields
#
#          be aweare of calling the proc in a db transaction
#          otherwise we will commit the changes in here!
#
# USAGE:    idref      identifier for unique key (name of field)
#          tablename  idref table name (default taf_idref)
#          start      idref start value (default db definition)
#          incr        idref increment value (default db definition)
#          limit      idref limit value (default db definition)
#
#          start, incr, limit are only considered for a new idref!
#
# RETURN:  key        generated key value
# ==========================================================
proc y_db_getId {idref {table {taf_idref}} {start {}} \
                      {incr {}} {limit {}} } {
    global errorInfo errorCode DB_lib_priv Fields

    # make sure we have the table def
    y_table_def $table

    # lock the table to make sure we are the only one fiddling

    switch -- $DB_lib_priv(DB_TYPE) {
        adab    {
            set lock "lock (wait) table $table in exclusive mode"
        }
        orac    {
            set lock "lock table $table in exclusive mode"
        }
    }

    set ret [sql run $lock ]
    Debug $ret 9

    # read the last value
    set cmd "select i_last, i_start, i_incr, i_limit from $table
                    where i_idref = '$idref'"
    # open cursor
    if [catch {sql open $cmd} c] {
        Debug "Error in sql open: $c error=$errorCode"
        if {$errorCode < 0} {
            # return error
            return -code error
        }
    }
    Debug curs=$c 9

    if {![string length $c]} {

        # no record found, insert it
        if {[string length $start]} {
            set i_start $start
        } else {
            set i_start $Fields(i_start.value)
        }
        set i_last $i_start

        if {[string length $incr]} {
            set i_incr $incr
        } else {
            set i_incr $Fields(i_incr.value)
        }

        if {[string length $limit]} {
            set i_limit $limit
        } else {
            set i_limit $Fields(i_limit.value)
        }

        set ins "insert into $table (i_idref,i_last,i_start,i_incr,i_limit) \
                        values ('$idref',$i_last,$i_start,$i_incr,$i_limit)"

        set ret [sql run $ins]
        if {$ret <= 0} {
            # return error
            return -code error -errorcode $ret
        }

    } else {
        # fetch record
        set row [sql fetch $c ]
        Debug $row 9
        set ret [sql close $c]

        array set r $row
        set i_last  $r(i_last)
        set i_start  $r(i_start)
        set i_incr  $r(i_incr)
        set i_limit  $r(i_limit)
    }

    # if freshly inserted
    if  { $i_last == 0 } {
        set i_last $i_start
    }

    # check limit
    if { $i_limit > 0 && $i_last >= $i_limit } {
        set i_last $i_start
    }

    # now increment last and update row on the fly
    incr i_last $i_incr

    set upd "update $table set i_last = $i_last \
                    where i_idref = '$idref'"

    set ret [sql run $upd]
    if {$ret <= 0} {
        # return error
        return -code error -errorcode $ret
    }

    # commit if not within transaction !
    if {$DB_lib_priv(tr_stat) == 0 } {
        set ret [sql run "COMMIT WORK"]
    }

    y_db_replication write $i_last

    return $i_last
}

For bookmarking purposes and shifting to new packages (or servers) simple text urls are important.

Certainly the cr style naming for forums etc. wouldn't go astray.

And Andrew: The problem came up when trying to move my personal web site to a fresh install (it currently runs very old code from way before there was even the faintest notion of an upgrade path),. I tried several tricks to move data so the object_id's would stay the same, but failed. It seemed like the number of objects created during install was larger than it used to, so some ID numbers would already be taken. In any event, it's tough.

Don's idea of creating a new sequence per package ID should work, though it's not terribly pretty.

I guess the prettiest solution is Dave's suggestion of creating a unique short string from the summary to use in the URL, though that doesn't solve the "Bug #123" problem.

/Lars

This is not really good in a high-concurrency environment.

One more argument:

Correct me if I'm wrong, but the latest code in Lars's new Bug Tracker package still has a race condition when creating these pretty "expose it in the URL" ids in bt_patch.new. That's both on the head and on the 4.6 branch, both Postgres and Oracle:

select nvl(max(patch_number),0) +1
into   v_patch_number
from   bt_patches
where  project_id = new.project_id;

I think we can all agree that Lars is one of our best and most skilled developers, yes? So if he got it wrong, what does that say about how likely other developers are to get it wrong?

Therefore, I suggest that if the, "No, we don't want to expose the object_id in URLs, we want something else" camp really wants their way, that a necessary (but not necessarily sufficient) requirement is that is someone first design and implement a general, scaleable, and correct solution that can be used by any package to map object_ids to some or one "pretty" id representation.

How 'bout we create our own internet unique 'tinyurl' like is available from TinyUrl.com. Of course ours would be free of ads and would be useable from a simple ns_geturl request. I don't see any other way to crack the bug than to have one single place on the internet to generate unique ids.

Maybe you could install djb's libtai and create an attosecond precision timestamp and convert that into a 16 byte packed representation. Maybe uuencode that. Apparently this would handle up to 1x10^18 objects per second.

Maybe you could request a series of numbers in one request.

Actually I guess a simple udp protocol would work great.