Content Manager 9.2 Nuggets

December 03, 2017 by Erik W in Configuration

Here's a collection of changes introduced with 9.2....

IDOL is gone, sort of

For the end-users absolutely nothing has changed. They will still search like they always have done. However, document content indexing should now be implemented with ElasticSearch. If you aren't on the bandwagon with that, you've probably invested in other uses for IDOL. You can still configure 9.2 to use IDOL if you need. However, for everyone else ElasticSearch is the future.

You can see that both are still referenced on the administration ribbon. In my environment I have no IDOL components, but the toolbar options are still there. I'll happily ignore the IDOL ones.

Elasticsearch build-out

You can quickly find yourself unsure how to resource this new technology. The course I took quickly rationalized 9 servers and over 128GB RAM for a database of just 1 million documents. Below is the diagram used for that example. Although entirely unrealistic I'm sure, it highlights what's meant by "elastic".

As time permits I'll post more about how to properly design an ElasticSearch infrastructure. KeyView is still used to generate the content for the engine, and the engine is managed via rest commands. However, unlike IDOL, there are really cool user interfaces for managing it (some at a minor cost).

Navigating via Keyboard

When working with tree views you can no longer use arrow keys to expand/collapse a tree. For example when trying to expand a category....

To expand this you can press + or -. Alternatively, you can press control + asterisk on the numeric keypad. Doing so will expand the structure recursively. As noted in the comments, a bug report has been raised for this issue.

Webclient Record Nesting

You can now expand a container and see the contents. You can even expand contents and continue drilling down.

Webclient Tabs

Tabs now appear as you perform actions. You could have a search result in one tab, a new location in another, and saved searches in a third. I do not see any option to have tabs auto-open at startup (like the thick client has).

Metadata Validation Rules

There's a new feature that aims to solve some of the validation challenges we have with metadata. You can find the feature off the administration ribbon...

Right-click->New to create a new one

Here's the configuration dialog...

Silly me tries something silly....

The help file says...

LookupSet Changes

Many new options with this one. Here's the property page for a "States" lookup set. Note how I've enabled both short and long values on the items.

With this configuration the lookup item property page looks like below...

Here's what a few entries look like with comments enabled...

Hierarchical Lookup Items

I created a lookup set name "Old Projects" and defined it as follows:

Then I created a few entries, with a few of those having sub-entries.

Here's how it appears to the end-users...

Error message shown if the user doesn't pick a "lowest level" item.

Once a valid value is selected, this is how it is saved into the field....

Lookup Items in Custom Properties

You can now pick Lookup Item when selecting the format of a custom property, as shown below...

Then you pick the lookup set from which the user will pick one item...

Then you give it a name...

For demonstration purposes I'll associate it with categories...

Before demonstrating it, I created another custom property of type string. I made this field use the same lookup set as its' source. Then I also associated it with the category object.

Next I selected Florida for both fields on an example category (the user experience was identical when selecting a value)...

After saving, I can see one major difference in the view pane....

The Lookup Item is a hyperlink whereas the string is just text.

Regardless which route you select, modifications to the Lookup Item are reflected across the dataset. In the screenshot below you can see how the selected state on the category shows a new value, even though I only updated the Lookup Item (and not the category).

Change is reflected even though I didn't modify category

SQL Database Changes

Schema Changes:

1 table removed, 3 views removed, 5 tables added, and two new encryption functions

Across the board it appears all VARCHAR type columns have now been converted to NVARCHAR. Below you can see some of the results of a schema comparison. I've highlighted the example column.

Check In Style Email Clean-up Option

In the previous version you could "Keep email in the mail system" and/or "Move deleted email to Deleted Items" (or neither). Now you can only select one of these three options: Permanent Delete, Move to Deleted Items, Retain in Mail System.

Check-in Styles Link off Administration Ribbon

Administrators can gain access to all check-in styles from here...

Classification/Category Specific Record Types

Limit which record types can be associated with classification/category...

If a user attempts to pick it they get an error...

Automatically Declare as Final

I would think this makes sense for email, but here's what the help says....

“If the document store is enabled for SEC compliance then this option will be enabled, see About SEC Rule 17a-4 Compliance rules. Whenever a record is created that has a document attached, the record will be automatically finalized at the time it is saved to the database. If you attach a document to an existing record that doesn’t have a document attached, then that record will be finalized. The options for creating revisions are prevented.”

Here's the option on the record type:

Default Copy Style for Referenced Access Controls

When creating a record type or classification you can define the behavior of referenced access controls. How you go about accessing the options has changed. Clicking copy style shows you a popup.

In the previous version you accessed it from a second tab...

Update Last Action Date when Viewing

In previous versions there was a global setting that controlled whether the last action date is to be updated when viewing a record. This concept has now been moved to the record type level. There is no longer a global setting for this behavior.

Confirm Preview of Document

When enabled the user must click anywhere within the preview window, confirming their intention to view the record. This was always a personal concern as an administrator may accidentally leave the preview pane enabled whilst they peruse search results (with no intention or desire to actually see the record).

Toggle this setting on the record type's audit property page

Feature Activation Required

If you attempt to enable the External Stores, Manage In Place, EMC Storage, or iTernity Content Addressable Storage features, you'll be prompted for an activation key for each...

Render to PDF after save

This one has been here for a few versions I guess, but I just noticed it...

Recreating the NARA JFK Assassination Archive within Content Manager

October 29, 2017 by Erik W in Configuration

This past week the National Archives released numerous records from their JFK Assassination collection. As I reviewed what they've released I found myself trying to imagine how it might be managed within Content Manager (they are using OpenText). What sort of configuration would facilitate a declassification/review/release process? How would I handle new legislation and its' impact on my records retention? I hope to address these thoughts, and a few more, over the coming posts.

Before digging into those advanced topics, I'm going to try and import the released records into a new instance of Content Manager. I started by visiting the 2017 release page for the collection. The page shows me the meta-data for the records, a link to download each, and a link to download an excel spreadsheet (containing all the meta-data).

I downloaded the spreadsheet and then opened it to take a look at the columns. I should be able to map each field to a property within Content Manager, or decide to not import it at all. A few of the fields will need to have lookup sets created to support them.

Spreadsheet fields and my thoughts on mapping each:

File Name -- URL of the electronic record, which will need to be retrieved and then attached to the CM record
Record Num -- the NARA record number which will also be used as the CM record number
NARA Release Date -- record date published
Formerly Withheld -- create a new custom string property with supporting lookup set
Agency -- create a new custom string property with supporting lookup set
Doc Date -- record date created
Doc Type -- create a new custom string property with supporting lookup set
File Num -- this is interesting because values refer to container titles (not container record numbers), so initially I'll map it to a CM custom string property and later determine what to do with containers.
To Name -- create a new custom string property
From Name -- create a new custom string property
Title -- record title
Num Pages -- create a new custom number property
Originator -- record owner
Record Series -- if containers are used then this series would most likely be indicated on that container, so for now I'll import the values into a new custom string property and tackle it in a later post.
Review Date -- create a new custom date property
Comments -- here I'll create a new custom string property. Although I could use the existing record notes property, there are behavioral settings at the record type level which impact user's abilities to interact with that field and therefore shouldn't be used relied upon (I don't think so at least).
Pages Released -- create a new custom number property

With the mapping details worked out I can move on to the configuration. I always step through configuration in the following order: security, locations, schedules, classifications, thesaurus terms (keywords), lookup sets, custom properties, record types and system settings. The remainder of this post will step through each and at the end I'll have a dataset with all of these records imported.

Terminology

Within the National Archives some CM concepts are referenced differently. For instance, Security Levels and Caveats are referred to as Security Classifications and Supplemental Markings respectively; so I decided to first change the terminology option on the compliance tab of the system options to "US Common".

Security

With my terminology now aligned with NARA, I created the standard security classification rankings. I'm not going to create any supplemental markings at the moment. Since I intend to demonstrate a CM declassification process it makes sense to have a few levels available as soon as possible, even if I don't use them during my initial import.

Locations

The filter option within my spreadsheet application allows me to see that there are a handful of values I could use as a location list. Now each of these locations should have a default security profile with the "Top Secret" security classification, as each could possibly have records at that level.

I have a couple of options with regard to locations:

Manually create each location, especially since there are less than twenty distinct originators in the spreadsheet.
Use a spreadsheet formula/function to export a new worksheet that will then be imported via DataPort's Spreadsheet Formatter
Save the spreadsheet off as a CSV file and then use powershell to create the locations (thereby giving me greater flexibility but also requiring some manual efforts).

I decided to go the last route because I can easily re-use what I create. I downloaded the workbook from NARA and converted it from Excel to CSV. I then wrote this powershell script to extract/create all of the unique originators:

Add-Type -Path "D:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$db = New-Object HP.HPTRIM.SDK.Database
$db.Connect
$metadataFile = "F:\Dropbox\CMRamble\JFK\nara_jfk_2017release.tsv"
$owners = Get-Content -Path $metadataFile | ConvertFrom-Csv -Delimiter "`t" | select "Originator" | sort-object -Property "Originator" -Unique
foreach ( $owner in $owners ) 
{
    if ( [String]::IsNullOrWhiteSpace($owner.Originator) -ne $true ) {
        $location = New-Object HP.HPTRIM.SDK.Location -ArgumentList $db
        $location.TypeOfLocation = [HP.HPTRIM.SDK.LocationType]::Organization
        $location.IsWithin = $true
        $location.SortName = $owner.Originator
        $location.Surname = $owner.Originator
        $location.SecurityString = "Top Secret"
        $location.Save()
        Write-Host "Created $($location.FullFormattedName)"
    }
}

Schedules, Classifications, Thesaurus Terms

I don't really know how I'll leverage these features, yet. So for now I don't really need to do anything with them.

Lookup sets

According to my field mapping I need three lookup sets: Formerly Withheld, Agency, and Doc Type. I could manually create each of these, use Data Port, or use a powershell script. Since I have a powershell script handy that can extract unique values from a column, I elect to go the powershell route.

Here is my powershell script that creates each of these three lookup sets and adds items for each unique value:

Add-Type -Path "D:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$db = New-Object HP.HPTRIM.SDK.Database
$db.Connect
$metadataFile = "F:\Dropbox\CMRamble\JFK\nara_jfk_2017release.tsv"
$metaData = Get-Content -Path $metadataFile | ConvertFrom-Csv -Delimiter "`t" 
$fields = @("Formerly Withheld", "Agency", "Doc Type")
foreach ( $field in $fields ) 
{
    $fieldItems = $metaData | select $($field) | sort-object -Property $($field) -Unique | select $($field)
    $lookupSet  = $null
    $lookupSet = New-Object HP.HPTRIM.SDK.LookupSet -ArgumentList $db, $field
    if ( $lookupSet -eq $null ) 
    { 
        $lookupSet = New-Object HP.HPTRIM.SDK.LookupSet -ArgumentList $db 
        $lookupSet.Name = $field
        $lookupSet.Save()
        Write-Host " Created '$($field)' set"
    } 
    foreach ( $item in $fieldItems ) 
    {
        $itemValue = $($item.$($field))
        $lookupItem = New-Object HP.HPTRIM.SDK.LookupItem -ArgumentList $db, $itemValue
        if ( $lookupItem.Error -ne $null ) 
        {
            $lookupItem = New-Object HP.HPTRIM.SDK.LookupItem -ArgumentList $lookupSet 
            $lookupItem.Name = $itemValue
            $lookupItem.Save()
            Write-Host " Added '$($itemValue)' item"
        }
		$lookupItem = $null
    }
	$lookupSet = $null
}

Here's what the results look like in my dataset...

Custom Properties

Next I created each of the required custom properties, as defined in the mapping notes at the top of this post. I, as usual, don't like doing this manually via the client. Powershell makes it very easy!

Here's my powershell script to create a field for every column in the spreadsheet. Note that I created custom properties for some of the columns that I've mapped to stock record properties (title for instance). I decided to store the original data on the record and then populate the stock properties as needed (which will be done for title, doc date, number).

Add-Type -Path "D:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$db = New-Object HP.HPTRIM.SDK.Database
$db.Connect
 
$lookupFields = @("NARA Formerly Withheld", "NARA Agency", "NARA Doc Type")
foreach ( $lookupField  in $lookupFields ) 
{
    $field = New-Object HP.HPTRIM.SDK.FieldDefinition $db
    $field.Name = $lookupField
    $field.LookupSet = New-Object HP.HPTRIM.SDK.LookupSet $db, $lookupField
    $field.Save()
 
    Write-Host "Created Property for Lookup Set '$($lookupField)'"
}
$stringFields = @("NARA File Num", "NARA To Name", "NARA From Name", "NARA Title", "NARA Originator", "NARA Record Series", "NARA Comments", "NARA Record Num", "NARA File Name")
foreach ( $stringField  in $stringFields ) 
{
    $field = New-Object HP.HPTRIM.SDK.FieldDefinition $db
    $field.Name = $stringField
    $field.Format = [HP.HPTRIM.SDK.UserFieldFormats]::String
    $field.Length = 50
    $field.Save()
 
    Write-Host "Created Property '$($stringField)' as String with length $($field.Length)"
}
$numberFields = @("NARA Num Pages", "NARA Pages Released")
foreach ( $numberField  in $numberFields ) 
{
    $field = New-Object HP.HPTRIM.SDK.FieldDefinition $db
    $field.Name = $numberField
    $field.Format = [HP.HPTRIM.SDK.UserFieldFormats]::Number
    $field.Save()
 
    Write-Host "Created Property '$($numberField)' as Number"
}
 
$dateFields = @("NARA Release Date", "NARA Doc Date", "NARA Review Date")
foreach ( $dateField in $dateFields ) 
{
    $field = New-Object HP.HPTRIM.SDK.FieldDefinition $db
    $field.Name = $dateField 
    $field.Format = [HP.HPTRIM.SDK.UserFieldFormats]::Date
    $field.Save()
 
    Write-Host "Created Property '$($dateField)' as Date"
}

Here's the output from the script...

Record Types

My dataset was prepopulated during creation because I elected to have the standard data template loaded (this was an option on the initialize page of the new dataset creation wizard). I can update the pre-created document record type so that it can support the records I want to import. During a later post I will work with folders/containers.

As the image below shows, I tagged all of the available custom properties on the document record type.

I also changed the numbering pattern to a bunch of x's. This will allow me to stuff any value into the record number field during import.

System Settings

At this point I'm ready to import the entire JFK archive. I don't need to worry about any of the system settings. I'll leave all of the default options (except terminology).

Import the Archive

In order for me to use DataPort, I would need to manipulate the data source. That's because I want to store two copies of several fields. For instance: the title. I wish to save the title column into both the record title and a custom property named "NARA Title". I could duplicate the column within the spreadsheet and then craft a DataPort project to map the two fields. I could also write a powershell script that manages these ETL (Extract, Transform, Load) tasks.

Here's the powershell script I crafted to import this spreadsheet (which was saved as a tab delimited file):

Add-Type -Path "D:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$db = New-Object HP.HPTRIM.SDK.Database
$db.Connect
$metadataFile = "F:\Dropbox\CMRamble\JFK\nara_jfk_2017release.tsv"
$metaData = Get-Content -Path $metadataFile | ConvertFrom-Csv -Delimiter "`t"
$doc = New-Object HP.HPTRIM.SDK.RecordType -ArgumentList $db, "Document"
$fields = @("Agency", "Comments", "Doc Date", "Doc Type", "File Name", "File Num", "Formerly Withheld", "From Name", "NARA Release Date", "Num Pages", "Originator", "Record Num", "Record Series", "To Name", "Review Date", "Title")
foreach ( $meta in $metaData ) 
{
    #download the record if needed
    $localFileName = "$PSScriptRoot\$($meta.'File Name')"
    if ( (Test-Path -Path $localFileName) -eq $false ) 
    {
        Write-Host "Downloading $($meta.'File Name')"
        [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
        $url = "https://www.archives.gov/files/research/jfk/releases/$(($meta.'File Name').ToLower())"
        $wc = New-Object System.Net.WebClient
        $wc.DownloadFile($url, $localFileName)
    }
    #check if record already exists
    $record = $null
    $existingObject = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch $db, Record
    $existingObject.SearchString = "number:$($meta.'Record Num')"
    if ( $existingObject.Count -eq 1 ) 
    {
        foreach ( $rec in $existingObject ) {
            $record = [HP.HPTRIM.SDK.Record]($rec)    
            break
        }
    } else {
        $record = New-Object HP.HPTRIM.SDK.Record -ArgumentList $db, $doc
        $record.LongNumber = $meta.'Record Num'
        $record.SetDocument($localFileName)
    }
    #correct title
    $record.TypedTitle = (&{If([String]::IsNullOrWhiteSpace($meta.Title)) { $meta.'File Name' } else { $meta.Title }})
    #attach document if necessary
    if ( $record.IsElectronic -eq $false ) {
        $record.SetDocument($localFileName)
    }
    #populate custom properties
    foreach ( $field in $fields ) 
    {
        if ( $field -eq "NARA Release Date" ) {
            $record.SetFieldValue((New-Object HP.HPTRIM.SDK.FieldDefinition -ArgumentList $db, "$($field)"), (New-Object HP.HPTRIM.SDK.UserFieldValue -ArgumentList $(($meta).$field)))
        } else {
            $record.SetFieldValue((New-Object HP.HPTRIM.SDK.FieldDefinition -ArgumentList $db, "NARA $($field)"), (New-Object HP.HPTRIM.SDK.UserFieldValue -ArgumentList $(($meta).$field)))
        }
    }
    $record.Save()
    Write-Host "Imported Record $($meta.'Record Num'): $($record.TypedTitle)"
}

Sweet! I executed the script and now I've got the full NARA JFK Archive housed within an instance of Content Manager!

My next goal with this archive: setup IDOL and analyze these documents! Stay tuned.

Security Breaches 101

October 18, 2017 by Erik W in Configuration

When I provide training for Content Manager I always start with security. It doesn't matter if security is a major concern for the organization or not. The main reason I do this, besides just needing to explain how it works, is to minimize the occurrences of security breaches. A security breach is a system flag indicating that the objects associated with a record are not in alignment with the security paradigm.

There are two places that you can define the behavior of security breaches within the software: the security tab of the system options and on the properties of a record type. If I use the out-of-the-box configuration of Content Manager, the former is set to "Display Warning" and the latter is set to "Ignore". For the moment I'll focus on the first.

Here's the option I'm talking about:

If you press F1 you'll see a description for the setting, shown below.

“When changing Assignee, Home or Owner for a Record to a less secure Location - set the method of Location change when a selected record has a lower security classification than the Location. See also Security breaches.

The options are
•Ignore
•Display Warning - default
•Prevent”

If I lookup "Security breaches", as it recommends, I get this description:

“This function enables a HPE Content Manager administrator to view and print occurrences of security breaches that may have occurred.

Security breaches appear in an Historical Events dialog box with all other events that have occurred, for example, Location and container changes and movement history.

By default, the security breaches function will only show security breaches for the current day.

For events to be logged, the relevant event must be selected in the Audit tab of the Record Type.”

The end result is a pop-up message like this:

Funny! I have the "Display warning" option selected but it still won't let me save it. I feel bad for developers at times... having to maintain this complex web of features and then having silly bugs like this. In a non-buggy build I would be able to click OK and continue on saving the record. For now I'll work around the bug by assigning it to myself and then reassigning it to Elmer.

Once I've done that I can go check out the online audit log. It will show me all the activity for today, including the right-most column that indicates if a security breach was detected. The screenshot below shows a series of assignments. Note the assignments to Elmer are breaches but the one to myself is not.

To resolve this I need to find out what's missing from Elmer's profile. Now you might think I could use the View Rights feature whilst impersonating Elmer, but you'd be wrong! As shown below, the UI is indicating the security profile is met by Elmer... but I suspect this is another bug

Instead, I look at the security profile of the record. I see that it has a security caveat of "HR"...

Now if I look at my location structure I can see the problem...

When I on-boarded Elmer I didn't give him a security profile. So it makes sense that assigning something to him would breach the security of the record. Elmer is completely unaware because he can't even access the record. So this notification is the administrator's opportunity to fix the problem. To resolve it I need to add the HR caveat to Elmer's profile.

The situation happens anytime you've elected to use a mixture of security levels and caveats. The best example is a secured facility's mail room, where they may be indexing incoming correspondence and assigning it to the wrong person accidentally. For that scenario they should be prevented from continuing, because we can't deliver the item as we register it officially. In other environments it's not the end of the world... it's something to manage.

If you take a look at this question over on the forum, you'll see a common situation. Imagine how much time he's going to have to spend fixing this problem! For a 50 user or less implementation it's a quick task; but for a 5,000 user site, good luck. He might want to engage a developer within his organization or a knowledgeable consultant.

I thought I'd be creative and write a powershell script to sort this out for him, but no such luck. It executes fine but doesn't actually save the updated security profile. Maybe a developer will read this and fix it! :)

Add-Type -Path "D:\Program Files\Hewlett Packard Enterprise\Content Manager\HP.HPTRIM.SDK.dll"
$db = New-Object HP.HPTRIM.SDK.Database
$db.Connect
$breaches = New-Object HP.HPTRIM.SDK.TrimMainObjectSearch -ArgumentList $db, History
$breaches.SearchString = "breach"
foreach ( $breach in $breaches ) 
{
    $location = ([HP.HPTRIM.SDK.History]$breach).MovementLocation
    $unit = $location.Unit
    $recordProfile = ([HP.HPTRIM.SDK.History]$breach).Record.SecurityProfile
    if ( $unit -ne $null -and $unit.SecurityProfile.CanAccess($recordProfile) ) {
        $location.SecurityProfile.UpgradeTo($unit.SecurityProfile)
        $location.Save()
        Write-Host "Updating $($location.FullFormattedName) to security profile of $($unit.FullFormattedName)"
    }
}

Executing this will at least give me a report of all the users that should be updated. The example below lists Elmer Fudd twice, exactly as shown in the online audit log.