Example Use Case - Configuring a Multizone Secure Data Store

The following steps walk you through the process of deploying six Storage Nodes across three primary zones. You can then verify that each shard has a replica in every zone; service can be continued in the event of a zone failure. You will configure secure data store in all the six Storage Nodes. In the first Storage Node, security will be configured and the security directory and all files contained in it will be copied from the first Storage Node to other Storage Nodes to setup security.

Follow the steps below in the first Storage Node(node01):
  • Execute the following command:
    java -jar $KVHOME/lib/kvstore.jar makebootconfig \
-root $KVROOT \
-port 5000 \
-host $KVHOST \
-harange 5010,5020 \
-store-security configure \
-capacity 1 \
-storagedir ${KVDATA}/disk1 \
-storagedirsize 5500-MB \
  • Start the Storage Node Agent:
    java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &
  • Create a zip file of all the security files created:
    cd ; 
zip -r $HOME/security.zip $KVROOT/security; 
cd -
  • Copy $HOME/security.zip from this node (node01 ) to the other five nodes.
Follow these steps in each of the other Storage Nodes (node02, node03, node04, node05, node06).
  • Unzip the security files copied from the first Storage Node (node01 ).
    cd; 
unzip -o security.zip -d /; 
cd -;
  • Execute the following command:
    java -jar $KVHOME/lib/kvstore.jar makebootconfig \
-root $KVROOT \
-port 5000 \
-host $KVHOST \
-harange 5010,5020 \
-store-security enable \
-capacity 1 \
-storagedir ${KVDATA}/disk1 \
-storagedirsize 5500-MB \
  • Start the Storage Node Agent:
    java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &
From the first Storage Node (node01 ) deploy your data store using the following steps:
  • Start the Admin CLI. Here $KVHOST is node01.
    java -Xmx64m -Xms64m \
-jar  $KVHOME/lib/kvstore.jar runadmin \
-port 5000 -host $KVHOST
-security $KVROOT/security/client.security
  • Name your data store and deploy three primary zones:
    configure -name MetroArea;
plan deploy-zone -name "Manhattan" -rf 1 -wait;
plan deploy-zone -name "JerseyCity" -rf 1 -wait;
plan deploy-zone -name "Queens" -rf 1 -wait;
  • Deploy the first Storage Node with administration process in the Manhattan zone.
    plan deploy-sn -znname Manhattan -host node01 -port 5000 -wait;
plan deploy-admin -sn sn1 -wait;
    Deploy a second Storage Node in Manhattan zone:
    plan deploy-sn -znname Manhattan -host node02 -port 5000 -wait;
  • Deploy the first Storage Node with administration process in the JerseyCity zone:.
    plan deploy-sn -znname JerseyCity -host node03 -port 5000 -wait;
plan deploy-admin -sn sn3 -wait;
    Deploy a second Storage Node in JerseyCity zone:
    plan deploy-sn -znname JerseyCity -host node04 -port 5000 -wait;
  • Deploy the first Storage Node with administration process in the Queens zone:.
    plan deploy-sn -znname Queens -host node05 -port 5000 -wait;
plan deploy-admin -sn sn5 -wait;
    Deploy a second Storage Node in Queens zone:
    plan deploy-sn -znname JerseyCity -host node06 -port 5000 -wait;
  • Create and deploy a topology: 
    topology create -name Topo1 -pool AllStorageNodes -partitions 300;
plan deploy-topology -name Topo1 -wait;
  • Create users and configure security with remote access using the following steps:
    • Create the first admin user and grant the required readwrite privileges
    • Store the password in a password file or wallet based on the password management mechanism that you have configured during the bootstrapping process.
    • Take a copy of the client.security file that is present in the security folder. Name the copied file (for example, root.login). Append the user credentials (user name and password file/wallet) created in the previous steps, as security properties in this file.
    • Copy the security folder to all the Storage Nodes in the cluster.
  • Check service status with the show topology command: 
    kv-> show topology
    Output:
    store=MetroArea  numPartitions=100 sequence=117
  zn: id=zn1 name=Manhattan repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false
  zn: id=zn2 name=JerseyCity repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false
  zn: id=zn3 name=Queens repFactor=1 type=PRIMARY 
   allowArbiters=false masterAffinity=false

  sn=[sn1] zn:[id=zn1 name=Manhattan] node01:5000 capacity=1 RUNNING
    [rg1-rn1] RUNNING
          No performance info available
  sn=[sn2] zn:[id=zn1 name=Manhattan] node02:5000 capacity=1 RUNNING
    [rg2-rn1] RUNNING
          No performance info available
  sn=[sn3] zn:[id=zn2 name=JerseyCity] node03:5000 capacity=1 RUNNING
    [rg1-rn2] RUNNING
          No performance info available
  sn=[sn4] zn:[id=zn2 name=JerseyCity] node04:5000 capacity=1 RUNNING
    [rg2-rn2] RUNNING
          No performance info available
  sn=[sn5] zn:[id=zn3 name=Queens] node05:5000 capacity=1 RUNNING
    [rg1-rn3] RUNNING
          No performance info available
  sn=[sn6] zn:[id=zn3 name=Queens] node06:5000 capacity=1 RUNNING
    [rg2-rn3] RUNNING
          No performance info available

  numShards=2
  shard=[rg1] num partitions=50
    [rg1-rn1] sn=sn1
    [rg1-rn2] sn=sn3
    [rg1-rn3] sn=sn5
  shard=[rg2] num partitions=50
    [rg2-rn1] sn=sn2
    [rg2-rn2] sn=sn4
    [rg2-rn3] sn=sn6
  • Verify that each shard has a replica in every zone: 
    kv-> verify configuration
    Output:
    Verify: starting verification of store MetroArea 
based upon topology sequence #117
100 partitions and 6 storage nodes
Time: 2024-04-05 10:41:15 UTC   Version: 24.1.11
See node01:
$KVROOT/MetroArea/log/MetroArea_{0..N}.log 
for progress messages
Verify: Shard Status: healthy:2 
writable-degraded:0 read-only:0 offline:0 total:2
Verify: Admin Status: healthy
Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0
Verify: Zone [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0 
maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: Zone [name=Queens id=zn3 type=PRIMARY allowArbiters=false 
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0 
maxDelayMillis:4 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on node01:5000
Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false 
masterAffinity=false]
Status: RUNNING   Ver: <release version> <time> UTC  
Build id: <build ID> Edition: Enterprise
Verify: 	Admin [admin1]		Status: RUNNING,MASTER
Verify: 	Rep Node [rg1-rn1]	Status: RUNNING,MASTER 
sequenceNumber:1,261 haPort:5011 available storage size:31 GB
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on node02:5000    
Zone: [name=Manhattan id=zn1 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: <release version> <time> UTC    
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: 	Rep Node [rg2-rn1]	Status: RUNNING,MASTER 
sequenceNumber:1,236 haPort:5012 available storage size:31 GB
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on node03:5000    
Zone: [name=JerseyCity id=zn2 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: <release version> <time> UTC  
Build id: <build ID> Edition: Enterprise
Verify: 	Admin [admin2]		Status: RUNNING,REPLICA
Verify: 	Rep Node [rg1-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:1,261 haPort:5011 available storage size:31 GB 
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn4 ==
Verify: Storage Node [sn4] on node04:5000    
Zone: [name=JerseyCity id=zn2 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: <release version> <time> UTC  
Build id: <build ID> Edition: Enterprise
Verify: 	Rep Node [rg2-rn2]	Status: RUNNING,REPLICA 
sequenceNumber:1,236 haPort:5012 available storage size:31 GB 
delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn5 ==
Verify: Storage Node [sn5] on node05:5000    
Zone: [name=Queens id=zn3 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: <release version> <time> UTC  
Build id: <build ID> Edition: Enterprise
Verify: 	Admin [admin3]		Status: RUNNING,REPLICA
Verify: 	Rep Node [rg1-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:1,261 haPort:5011 available storage size:31 GB 
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn6 ==
Verify: Storage Node [sn6] on node06:5000    
Zone: [name=Queens id=zn3 type=PRIMARY 
allowArbiters=false masterAffinity=false]    
Status: RUNNING   Ver: <release version> <time> UTC  
Build id: <build ID> Edition: Enterprise
Verify: 	Rep Node [rg2-rn3]	Status: RUNNING,REPLICA 
sequenceNumber:1,236 haPort:5012 available storage size:31 GB 
delayMillis:4 catchupTimeSecs:0

Verification complete, no violations.

In the above example there are three zones (zn1 = Manhattan, zn2 = JerseyCity, zn3=Queens) with six replication nodes (two masters and four replicas) in the data store. This means that this topology is not only highly available because you have three replicas within each shard, but it is also able to recover from a single zone failure. If any zone fails, the other two zones are enough to elect the new master, so service continues without any interruption.

Adding Secondary Zone

The following example adds a secondary zone in a different geographical location, Europe, allowing the users to read the data from the secondary zone either because it is physically located closer to the client or because the primary zone in the New York metro area is unavailable due to a disaster. The steps involve creating and starting two new Storage Nodes with capacity 1, creating a secondary zone, deploying the new Storage Nodes in the secondary zone, and doing a redistribute of the topology so that a replica for each shard is placed in the secondary zone.

Follow these steps in both the new Storage Nodes (node07 and node08).
  1. Copy the security zipped files from the first node and unzip the files.
    unzip -o security.zip -d /;
  2. Invoke the makebootconfig utility for the first new Storage Node that will be deployed in the Frankfurt zone. The security configuration will be enabled while invoking the makebootconfig utility.
    java -jar $KVHOME/lib/kvstore.jar makebootconfig \
-root $KVROOT \
-port 5000 \
-host $KVHOST \
-harange 5010,5020 \
-store-security enable \
-capacity 1 \
-storagedir ${KVDATA}/disk1 \
-storagedirsize 5500-MB
  3. Start the Storage Node Agent. 
    java -jar $KVHOME/lib/kvstore.jar start -root $KVROOT &

To create a secondary zone and deploy the new Storage Nodes, do the following steps:

  1. Start the Admin CLI. Here $KVHOST is node01.
    java -Xmx64m -Xms64m \
-jar  $KVHOME/lib/kvstore.jar runadmin \
-port 5000 -host $KVHOST
-security $KVROOT/security/client.security
  2. Create a secondary zone in Frankfurt. 
    kv-> plan deploy-zone -name Frankfurt -rf 1 -type secondary -wait
    Output:
    Executed plan 14, waiting for completion...
Plan 14 ended successfully
  3. Deploy Storage Node sn7 in the Frankfurt zone.
    kv-> plan deploy-sn -znname Frankfurt -host node07 -port 5000 -wait
    Output:
    Executed plan 15, waiting for completion...
Plan 15 ended successfully
  4. Deploy the Storage Node sn7 with administration process in the Frankfurt zone.
    kv-> plan deploy-admin -sn sn7 -wait
    Output:
    Executed plan 16, waiting for completion...
Plan 16 ended successfully
  5. Deploy Storage Node sn8 in the Frankfurt zone.
    kv-> plan deploy-sn -znname Frankfurt -host node08  -port 5000 -wait
    Output:
    Executed plan 17, waiting for completion...
Plan 17 ended successfully
  6. Do redistribute and then deploy the new topology to create one replica for every shard in the secondary Frankfurt zone.
    kv-> topology clone -current -name topo_secondary
    Output:
    Created topo_secondary
    kv-> topology redistribute -name topo_secondary -pool AllStorageNodes
    Output:
    Redistributed: topo_secondary
    kv-> topology preview -name topo_secondary
    Output:
    Topology transformation from current deployed topology to topo_secondary:
Create 2 RN shard rg1 1 new RN : rg1-rn4
            shard rg2 1 new RN : rg2-rn4
    kv-> plan deploy-topology -name topo_secondary -wait
    Output:
    Executed plan 19, waiting for completion...
Plan 19 ended successfully
  7. Create users and configure security with remote access using the following steps:
    • Create the first admin user and grant the required readwrite privileges
    • Store the password in a password file or wallet based on the password management mechanism that you have configured during the bootstrapping process.
    • Take a copy of the client.security file that is present in the security folder. Name the copied file (for example, root.login). Append the user credentials (user name and password file/wallet) created in the previous steps, as security properties in this file.
    • Copy the security folder to all the Storage Nodes in the cluster.
  8. Check service status with the show topology command.
    kv-> show topology
    Output:
    store=MetroArea  numPartitions=100 sequence=120
  zn: id=zn1 name=Manhattan repFactor=1 type=PRIMARY
    allowArbiters=false masterAffinity=false
  zn: id=zn2 name=JerseyCity repFactor=1 type=PRIMARY
    allowArbiters=false masterAffinity=false
  zn: id=zn3 name=Queens repFactor=1 type=PRIMARY
    allowArbiters=false masterAffinity=false
  zn: id=zn4 name=Frankfurt repFactor=1 type=SECONDARY
    allowArbiters=false masterAffinity=false
 
  sn=[sn1] zn:[id=zn1 name=Manhattan] node01:5000 capacity=1 RUNNING
    [rg1-rn1] RUNNING
    single-op avg latency=0.21372496 ms multi-op avg latency=0.0 ms
  sn=[sn2] zn:[id=zn1 name=Manhattan] node02:5000 capacity=1 RUNNING
    [rg2-rn1] RUNNING
    single-op avg latency=0.30840763 ms multi-op avg latency=0.0 ms
  sn=[sn3] zn:[id=zn2 name=JerseyCity] node03:5000 capacity=1 RUNNING
    [rg1-rn2] RUNNING
    No performance info available
  sn=[sn4] zn:[id=zn2 name=JerseyCity] node04:5000 capacity=1 RUNNING
    [rg2-rn2] RUNNING
    No performance info available
  sn=[sn5] zn:[id=zn3 name=Queens] node05:5000 capacity=1 RUNNING
    [rg1-rn3] RUNNING
    No performance info available
  sn=[sn6] zn:[id=zn3 name=Queens] node06:5000 capacity=1 RUNNING
    [rg2-rn3] RUNNING
    No performance info available
  sn=[sn7] zn:[id=zn4 name=Frankfurt] node07:5000 capacity=1 RUNNING
    [rg1-rn4] RUNNING
    No performance info available  
  sn=[sn8] zn:[id=zn4 name=Frankfurt] node07:5000 capacity=1 RUNNING     
    [rg2-rn4] RUNNING     
    No performance info available
 
  numShards=2
  shard=[rg1] num partitions=50
    [rg1-rn1] sn=sn1
    [rg1-rn2] sn=sn3
    [rg1-rn3] sn=sn5
    [rg1-rn4] sn=sn7
  shard=[rg2] num partitions=50
    [rg2-rn1] sn=sn2
    [rg2-rn2] sn=sn4
    [rg2-rn3] sn=sn6
    [rg2-rn4] sn=sn8
  9. Verify that the secondary zone has a replica for each shard. 
    kv-> verify configuration
    Output:
    Verify: starting verification of store MetroArea
based upon topology sequence #120
100 partitions and 7 storage nodes
Time: <time> UTC   Version: <release version>
See node01:
$KVROOT/Disk1/MetroArea/log/MetroArea_{0..N}.log
for progress messages
Verify: Shard Status: healthy:2
writable-degraded:0 read-only:0 offline:0 total:2
Verify: Admin Status: healthy
Verify: Zone [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0
Verify: Zone [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0
maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: Zone [name=Queens id=zn3 type=PRIMARY allowArbiters=false
masterAffinity=false]   RN Status: online:2 read-only:0 offline:0
maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: Zone [name=Frankfurt id=zn4 type=SECONDARY allowArbiters=false
masterAffinity=false]   RN Status: online:1 read-only:0 offline:0
maxDelayMillis:1 maxCatchupTimeSecs:0
Verify: == checking storage node sn1 ==
Verify: Storage Node [sn1] on node01:5000   
Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING   Ver: <release version> <time> UTC
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Admin [admin1]        Status: RUNNING,MASTER
Verify:     Rep Node [rg1-rn1]    Status: RUNNING,MASTER
sequenceNumber:1,261 haPort:5011 available storage size:31 GB
Verify: == checking storage node sn2 ==
Verify: Storage Node [sn2] on node02:5000   
Zone: [name=Manhattan id=zn1 type=PRIMARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING   Ver: <release version> <time> UTC
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Rep Node [rg2-rn1]    Status: RUNNING,MASTER
sequenceNumber:1,236 haPort:5012 available storage size:31 GB
Verify: == checking storage node sn3 ==
Verify: Storage Node [sn3] on node03:5000   
Zone: [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING   Ver: <release version> <time> UTC
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Admin [admin2]        Status: RUNNING,REPLICA
Verify:     Rep Node [rg1-rn2]    Status: RUNNING,REPLICA
sequenceNumber:1,261 haPort:5011 available storage size:31 GB
delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn4 ==
Verify: Storage Node [sn4] on node04:5000   
Zone: [name=JerseyCity id=zn2 type=PRIMARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING   Ver: <release version> <time> UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Rep Node [rg2-rn2]    Status: RUNNING,REPLICA
sequenceNumber:1,236 haPort:5012 available storage size:31 GB
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn5 ==
Verify: Storage Node [sn5] on node05:5000   
Zone: [name=Queens id=zn3 type=PRIMARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING   Ver: <release version> <time> UTC
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Admin [admin3]        Status: RUNNING,REPLICA
Verify:     Rep Node [rg1-rn3]    Status: RUNNING,REPLICA
sequenceNumber:1,261 haPort:5011 available storage size:31 GB
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn6 ==
Verify: Storage Node [sn6] on node06:5000   
Zone: [name=Queens id=zn3 type=PRIMARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING  Ver: <release version> <time> UTC
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Rep Node [rg2-rn3]    Status: RUNNING,REPLICA
sequenceNumber:1,236 haPort:5012 available storage size:31 GB
delayMillis:0 catchupTimeSecs:0
Verify: == checking storage node sn7 ==
Verify: Storage Node [sn7] on node07:5000   
Zone: [name=Frankfurt id=zn4 type=SECONDARY allowArbiters=false
masterAffinity=false]   
Status: RUNNING   Ver: <release version> <time> UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify:     Admin [admin4]        Status: RUNNING,REPLICA 
Verify:     Rep Node [rg1-rn4]    Status: RUNNING,REPLICA
sequenceNumber:1,261 haPort:5011 available storage size:31 GB
delayMillis:1 catchupTimeSecs:0
Verify: == checking storage node sn8 ==
Verify: Storage Node [sn8] on node08:5000 
Zone: [name=Frankfurt id=zn4 type=SECONDARY allowArbiters=false 
masterAffinity=false] 
Status: RUNNING Ver: <release version> <time> UTC 
Build id: c8998e4a8aa5 Edition: Enterprise
Verify: Rep Node [rg2-rn4] Status: RUNNING,REPLICA 
sequenceNumber:1,238 haPort:5012 available storage size:31 GB 
delayMillis:0 catchupTimeSecs:0 

Verification complete, no violations.