1. What is Temporal?

2. Basic Architecture

3. Senior Staff Engineer Design for Small Startup/MVP

4. Implementation

5. Deployment & Cost Optimization

6. Client Connection & Job Management

7. Monitoring & Operations

Temporal is a durable execution platform that enables developers to write fault-tolerant applications that can survive failures, restarts, and infrastructure issues. Temporal was not an overnight idea. It took at least 15 years to get to where we are.

1. Workflows: With Temporal, those steps are defined by writing code, known as a Workflow Definition, and are carried out by running that code, which results in a Workflow Execution. Workflows are resilient functions that maintain their state even across failures.

2. Activities: Individual units of work (like API calls, database operations) that can fail and be retried independently.

3. Workers: Workers are compute nodes that run your Temporal application code. You compile your workflows and activities into a worker executable.

4. Task Queues: Communication mechanism between the Temporal service and workers.

• Automatic retries: Handle Gmail API rate limits and network failures gracefully

• Long-running operations: Temporal Workflows are resilient. They can run—and keeping running—for years, even if the underlying infrastructure fails.

• State management: Track sync progress, handle partial syncs

• Visibility: Built-in UI to monitor and debug sync operations

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Client App    │────▶│ Temporal Service │◀────│    Workers      │
│  (API/Web/CLI)  │     │  (Cloud/Self)    │     │ (Your Code)     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                 │
                                 ▼
                        ┌─────────────────┐
                        │   Data Store    │
                        │ (MySQL/Cassandra)│
                        └─────────────────┘

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   REST API      │────▶│ Temporal Client  │────▶│ Temporal Cloud  │
│  (Express.js)   │     │                  │     │                 │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                           │
                                                           ▼
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Gmail API      │◀────│ Sync Activities  │◀────│ Sync Workflow   │
│                 │     │                  │     │                 │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                 │
                                 ▼
                        ┌─────────────────┐
                        │   PostgreSQL    │
                        │ (Message Store) │
                        └─────────────────┘

1. Simple & Extensible: Start with core sync functionality, design for future features

2. Cost-Effective: Minimize infrastructure costs while maintaining reliability

3. Observable: Built-in monitoring and debugging capabilities

4. Scalable: Handle growth from 10 to 10,000 users without major refactoring

// Domain Model
interface User {
  id: string;
  email: string;
  gmailRefreshToken: string;
  lastSyncHistoryId?: string;
  syncSettings: SyncSettings;
}

interface SyncSettings {
  syncFrequency: 'realtime' | 'hourly' | 'daily';
  labelsToSync: string[];
  maxMessagesPerSync: number;
}

interface SyncResult {
  messagesAdded: number;
  messagesModified: number;
  messagesDeleted: number;
  newHistoryId: string;
  errors: SyncError[];
}

// Main Sync Workflow
export async function userGmailSyncWorkflow(
  userId: string
): Promise<void> {
  // Long-running workflow that manages all sync operations for a user
  
  // 1. Initial full sync if needed
  const user = await activities.getUser(userId);
  if (!user.lastSyncHistoryId) {
    await activities.performFullSync(userId);
  }
  
  // 2. Set up recurring sync based on user preferences
  while (true) {
    try {
      // Check for Gmail push notifications or poll
      const hasChanges = await activities.checkForChanges(userId);
      
      if (hasChanges) {
        const syncResult = await activities.performIncrementalSync(userId);
        
        // Handle sync results
        if (syncResult.errors.length > 0) {
          await activities.notifyUser(userId, syncResult.errors);
        }
      }
      
      // Sleep based on sync frequency
      const sleepDuration = getSleepDuration(user.syncSettings.syncFrequency);
      await sleep(sleepDuration);
      
    } catch (error) {
      // Temporal will automatically retry based on retry policy
      console.error('Sync error:', error);
      await sleep('5m'); // Back off on errors
    }
  }
}

gmail-sync-server/
├── src/
│   ├── workflows/
│   │   ├── gmail-sync.workflow.ts
│   │   └── index.ts
│   ├── activities/
│   │   ├── gmail.activities.ts
│   │   ├── database.activities.ts
│   │   └── index.ts
│   ├── worker/
│   │   └── index.ts
│   ├── api/
│   │   ├── server.ts
│   │   └── routes/
│   ├── client/
│   │   └── temporal.client.ts
│   └── shared/
│       └── types.ts
├── docker-compose.yml
├── package.json
└── tsconfig.json

# Create project
mkdir gmail-sync-server
cd gmail-sync-server
npm init -y

# Install dependencies
npm install @temporalio/client @temporalio/worker @temporalio/workflow @temporalio/activity @temporalio/common
npm install express @types/express
npm install googleapis @types/node
npm install pg @types/pg
npm install --save-dev typescript @types/node ts-node nodemon

// src/activities/gmail.activities.ts
import { google } from 'googleapis';
import { OAuth2Client } from 'google-auth-library';

export async function getGmailClient(refreshToken: string): Promise<any> {
  const oauth2Client = new OAuth2Client(
    process.env.GMAIL_CLIENT_ID,
    process.env.GMAIL_CLIENT_SECRET,
    process.env.GMAIL_REDIRECT_URL
  );
  
  oauth2Client.setCredentials({
    refresh_token: refreshToken
  });
  
  return google.gmail({ version: 'v1', auth: oauth2Client });
}

export async function performFullSync(userId: string): Promise<SyncResult> {
  const user = await getUserFromDB(userId);
  const gmail = await getGmailClient(user.gmailRefreshToken);
  
  const result: SyncResult = {
    messagesAdded: 0,
    messagesModified: 0,
    messagesDeleted: 0,
    newHistoryId: '',
    errors: []
  };
  
  try {
    // Call messages.list to retrieve the first page of message IDs.
    let pageToken: string | undefined;
    
    do {
      const response = await gmail.users.messages.list({
        userId: 'me',
        maxResults: 100,
        pageToken,
        labelIds: user.syncSettings.labelsToSync
      });
      
      if (response.data.messages) {
        // Process messages in batches
        const messageIds = response.data.messages.map(m => m.id);
        await processBatchMessages(gmail, userId, messageIds);
        result.messagesAdded += messageIds.length;
      }
      
      pageToken = response.data.nextPageToken;
      
      // Store the historyId for future incremental syncs
      if (response.data.historyId) {
        result.newHistoryId = response.data.historyId;
      }
      
    } while (pageToken && result.messagesAdded < user.syncSettings.maxMessagesPerSync);
    
    // Update user's lastSyncHistoryId
    await updateUserSyncState(userId, result.newHistoryId);
    
  } catch (error) {
    result.errors.push({
      type: 'FULL_SYNC_ERROR',
      message: error.message,
      timestamp: new Date()
    });
  }
  
  return result;
}

export async function performIncrementalSync(userId: string): Promise<SyncResult> {
  const user = await getUserFromDB(userId);
  const gmail = await getGmailClient(user.gmailRefreshToken);
  
  const result: SyncResult = {
    messagesAdded: 0,
    messagesModified: 0,
    messagesDeleted: 0,
    newHistoryId: user.lastSyncHistoryId || '',
    errors: []
  };
  
  try {
    // you can perform a partial sync using the history.list method to return all history records newer than the startHistoryId you specify in your request.
    const historyResponse = await gmail.users.history.list({
      userId: 'me',
      startHistoryId: user.lastSyncHistoryId,
      labelId: user.syncSettings.labelsToSync
    });
    
    if (historyResponse.data.history) {
      for (const historyRecord of historyResponse.data.history) {
        // Process each history record
        if (historyRecord.messagesAdded) {
          result.messagesAdded += historyRecord.messagesAdded.length;
        }
        if (historyRecord.messagesDeleted) {
          result.messagesDeleted += historyRecord.messagesDeleted.length;
        }
        if (historyRecord.labelsAdded || historyRecord.labelsRemoved) {
          result.messagesModified += 1;
        }
      }
    }
    
    if (historyResponse.data.historyId) {
      result.newHistoryId = historyResponse.data.historyId;
      await updateUserSyncState(userId, result.newHistoryId);
    }
    
  } catch (error) {
    if (error.code === 404) {
      // If the startHistoryId supplied by your client is outside the available range of history records, the API returns an HTTP 404 error response.
      // Need to perform full sync
      return performFullSync(userId);
    }
    
    result.errors.push({
      type: 'INCREMENTAL_SYNC_ERROR',
      message: error.message,
      timestamp: new Date()
    });
  }
  
  return result;
}

// src/workflows/gmail-sync.workflow.ts
import * as workflow from '@temporalio/workflow';
import type * as activities from '../activities';

// Import activity types
const { 
  performFullSync, 
  performIncrementalSync, 
  checkForChanges,
  getUser,
  notifyUser 
} = workflow.proxyActivities<typeof activities>({
  startToCloseTimeout: '5m',
  retry: {
    initialInterval: '30s',
    backoffCoefficient: 2,
    maximumInterval: '10m',
    maximumAttempts: 5
  }
});

// Define signals for external control
export const pauseSyncSignal = workflow.defineSignal('pauseSync');
export const resumeSyncSignal = workflow.defineSignal('resumeSync');
export const updateSettingsSignal = workflow.defineSignal<[SyncSettings]>('updateSettings');

export async function userGmailSyncWorkflow(
  userId: string
): Promise<void> {
  let isPaused = false;
  let syncSettings: SyncSettings | null = null;
  
  // Set up signal handlers
  workflow.setHandler(pauseSyncSignal, () => {
    isPaused = true;
  });
  
  workflow.setHandler(resumeSyncSignal, () => {
    isPaused = false;
  });
  
  workflow.setHandler(updateSettingsSignal, (newSettings) => {
    syncSettings = newSettings;
  });
  
  // Main sync loop
  while (true) {
    if (isPaused) {
      await workflow.sleep('1m');
      continue;
    }
    
    try {
      const user = await getUser(userId);
      if (!user) {
        throw new Error(`User ${userId} not found`);
      }
      
      // Use latest settings
      if (syncSettings) {
        user.syncSettings = syncSettings;
      }
      
      // Perform initial full sync if needed
      if (!user.lastSyncHistoryId) {
        await performFullSync(userId);
      }
      
      // Check for changes
      const hasChanges = await checkForChanges(userId);
      
      if (hasChanges) {
        const syncResult = await performIncrementalSync(userId);
        
        if (syncResult.errors.length > 0) {
          await notifyUser(userId, syncResult.errors);
        }
      }
      
      // Sleep based on sync frequency
      const sleepDuration = getSleepDuration(user.syncSettings.syncFrequency);
      await workflow.sleep(sleepDuration);
      
    } catch (error) {
      console.error(`Sync error for user ${userId}:`, error);
      // Temporal will retry based on the retry policy
      await workflow.sleep('5m'); // Back off on errors
    }
  }
}

function getSleepDuration(frequency: string): string {
  switch (frequency) {
    case 'realtime':
      return '1m';
    case 'hourly':
      return '1h';
    case 'daily':
      return '24h';
    default:
      return '1h';
  }
}

// src/worker/index.ts
import { Worker, NativeConnection } from '@temporalio/worker';
import * as activities from '../activities';

async function run() {
  // Your first development environment will probably only need one worker that runs both workflows and activities in a single process.
  
  const connection = await NativeConnection.connect({
    address: process.env.TEMPORAL_ADDRESS || 'localhost:7233'
  });

  const worker = await Worker.create({
    connection,
    namespace: process.env.TEMPORAL_NAMESPACE || 'default',
    taskQueue: 'gmail-sync-queue',
    workflowsPath: require.resolve('../workflows'),
    activities,
  });

  await worker.run();
}

run().catch((err) => {
  console.error(err);
  process.exit(1);
});

// src/api/server.ts
import express from 'express';
import { Connection, Client } from '@temporalio/client';
import { userGmailSyncWorkflow } from '../workflows/gmail-sync.workflow';

const app = express();
app.use(express.json());

let temporalClient: Client;

async function getTemporalClient(): Promise<Client> {
  if (!temporalClient) {
    const connection = await Connection.connect({
      address: process.env.TEMPORAL_ADDRESS || 'localhost:7233'
    });
    
    temporalClient = new Client({
      connection,
      namespace: process.env.TEMPORAL_NAMESPACE || 'default'
    });
  }
  
  return temporalClient;
}

// Start sync for a user
app.post('/api/users/:userId/sync/start', async (req, res) => {
  try {
    const { userId } = req.params;
    const client = await getTemporalClient();
    
    // Although it is not required, we recommend providing your own Workflow Idthat maps to a business process or business entity identifier, such as an order identifier or customer identifier.
    const workflowId = `gmail-sync-${userId}`;
    
    const handle = await client.workflow.start(userGmailSyncWorkflow, {
      workflowId,
      taskQueue: 'gmail-sync-queue',
      args: [userId],
      // Workflow should run indefinitely
      workflowExecutionTimeout: undefined,
    });
    
    res.json({
      success: true,
      workflowId: handle.workflowId,
      runId: handle.firstExecutionRunId
    });
    
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

// Pause sync
app.post('/api/users/:userId/sync/pause', async (req, res) => {
  try {
    const { userId } = req.params;
    const client = await getTemporalClient();
    
    const workflowId = `gmail-sync-${userId}`;
    const handle = client.workflow.getHandle(workflowId);
    
    await handle.signal('pauseSync');
    
    res.json({ success: true, message: 'Sync paused' });
    
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

// Update sync settings
app.put('/api/users/:userId/sync/settings', async (req, res) => {
  try {
    const { userId } = req.params;
    const settings = req.body;
    const client = await getTemporalClient();
    
    const workflowId = `gmail-sync-${userId}`;
    const handle = client.workflow.getHandle(workflowId);
    
    await handle.signal('updateSettings', settings);
    
    res.json({ success: true, settings });
    
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

// Check sync status
app.get('/api/users/:userId/sync/status', async (req, res) => {
  try {
    const { userId } = req.params;
    const client = await getTemporalClient();
    
    const workflowId = `gmail-sync-${userId}`;
    const handle = client.workflow.getHandle(workflowId);
    
    const description = await handle.describe();
    
    res.json({
      status: description.status.name,
      startTime: description.startTime,
      runId: description.runId
    });
    
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {
  console.log(`API server listening on port ${PORT}`);
});

Since Temporal: Workflows run on workers that live on compute that you manage. This could mean a server you own and operate, a Kubernetes cluster hosted in EKS or your data center, or even ECS Fargate., we'll use ECS Fargate for a cost-effective deployment.

🎉 Startup Programs Available:

• General Startup Program: Our startup program offers $6,000 in free Temporal Cloud credits to startups with less than $30M in funding.

• AWS Activate Members: Temporal has partnered with AWS Activate to provide $3,000 in free Temporal Cloud credits

• Google Cloud Startups: This exclusive offer for members of the Google for Startups Cloud Program (GFSCP) provides $12,000 in free Temporal Cloud credits over 12 months

• New Users: $1,000 in free credits for new self-service sign-ups

Pricing Model: Actions pricing starts at $50 per million Actions ($0.00005 per Action). You gain progressive volume discounts as you scale.

Estimated Costs for Gmail Sync:

• 1,000 users × 24 syncs/day × 30 days = 720,000 syncs/month

• Each sync ≈ 10 actions (workflow start, activities, timers)

• Total: ~7.2M actions/month = $360/month

• Storage: storage is generally a very small percentage (3-5%) of the total bill.

• Support: the minimum support fee is $200 per month.

Total: ~$600/month for 1,000 active users (potentially covered by startup credits for 10+ months!)

Infrastructure Needs:

# Minimal Production Setup
services:
  temporal-frontend:
    cpu: 512
    memory: 1024
    desiredCount: 2
  
  temporal-history:
    cpu: 1024
    memory: 2048
    desiredCount: 2
  
  temporal-matching:
    cpu: 512
    memory: 1024
    desiredCount: 2
  
  temporal-worker-service:
    cpu: 512
    memory: 1024
    desiredCount: 1
  
  # Your workers
  gmail-sync-workers:
    cpu: 1024
    memory: 2048
    desiredCount: 2-10 # Auto-scale based on load

Estimated Costs:

• ECS Fargate: ~$300-500/month

• RDS MySQL (db.t3.small): ~$30/month

• ALB: ~$20/month

• Data transfer: ~$50/month

Total: ~$400-600/month

// temporal-worker-task-definition.json
{
  "family": "gmail-sync-worker",
  "networkMode": "awsvpc",
  "requiresCompatibilities": ["FARGATE"],
  "cpu": "1024",
  "memory": "2048",
  "containerDefinitions": [
    {
      "name": "worker",
      "image": "your-ecr-repo/gmail-sync-worker:latest",
      "environment": [
        {
          "name": "TEMPORAL_ADDRESS",
          "value": "temporal-cloud-address:7233"
        },
        {
          "name": "TEMPORAL_NAMESPACE",
          "value": "default"
        },
        {
          "name": "DATABASE_URL",
          "value": "postgresql://..."
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "/ecs/gmail-sync-worker",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ]
}

1. Use Fargate Spot for Workers: For lower level environments, such as test, engineering, or whatever your nomenclature is, it might not make sense to have tasks running all the time.

// ECS Service configuration
{
  "capacityProviderStrategy": [
    {
      "capacityProvider": "FARGATE_SPOT",
      "weight": 80,
      "base": 0
    },
    {
      "capacityProvider": "FARGATE",
      "weight": 20,
      "base": 2
    }
  ]
}

1. Auto-scaling Configuration:

// Auto-scaling policy
{
  "targetTrackingScalingPolicies": [
    {
      "targetValue": 70.0,
      "predefinedMetricSpecification": {
        "predefinedMetricType": "ECSServiceAverageCPUUtilization"
      },
      "scaleOutCooldown": 60,
      "scaleInCooldown": 300
    }
  ]
}

1. Schedule-based Scaling:

// Scale down during off-hours
{
  "scheduledActions": [
    {
      "scheduledActionName": "scale-down-night",
      "schedule": "cron(0 22 * * ? *)",
      "scalableTargetAction": {
        "minCapacity": 1,
        "maxCapacity": 2
      }
    },
    {
      "scheduledActionName": "scale-up-morning",
      "schedule": "cron(0 6 * * ? *)",
      "scalableTargetAction": {
        "minCapacity": 2,
        "maxCapacity": 10
      }
    }
  ]
}

# Dockerfile
FROM node:18-alpine

WORKDIR /app

# Install dependencies
COPY package*.json ./
RUN npm ci --only=production

# Copy application code
COPY . .

# Build TypeScript
RUN npm run build

# Run the worker
CMD ["node", "dist/worker/index.js"]

// Example: React Web App
import { useEffect, useState } from 'react';

function GmailSyncManager({ userId }) {
  const [syncStatus, setSyncStatus] = useState(null);
  
  const startSync = async () => {
    const response = await fetch(`/api/users/${userId}/sync/start`, {
      method: 'POST'
    });
    const data = await response.json();
    setSyncStatus(data);
  };
  
  const pauseSync = async () => {
    await fetch(`/api/users/${userId}/sync/pause`, {
      method: 'POST'
    });
  };
  
  const updateSettings = async (settings) => {
    await fetch(`/api/users/${userId}/sync/settings`, {
      method: 'PUT',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(settings)
    });
  };
  
  return (
    <div>
      <button onClick={startSync}>Start Sync</button>
      <button onClick={pauseSync}>Pause Sync</button>
      {/* Sync settings UI */}
    </div>
  );
}

// Handle Gmail push notifications
app.post('/api/webhooks/gmail', async (req, res) => {
  const { emailAddress, historyId } = req.body;
  
  // Find user by email
  const user = await findUserByEmail(emailAddress);
  
  if (user) {
    const client = await getTemporalClient();
    const workflowId = `gmail-sync-${user.id}`;
    
    try {
      const handle = client.workflow.getHandle(workflowId);
      
      // Signal the workflow that changes are available
      await handle.signal('gmailChangesAvailable', { historyId });
      
    } catch (error) {
      // Workflow might not be running, start it
      await client.workflow.start(userGmailSyncWorkflow, {
        workflowId,
        taskQueue: 'gmail-sync-queue',
        args: [user.id]
      });
    }
  }
  
  res.status(200).send();
});

// Add OpenTelemetry tracing
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { OpenTelemetryActivityInboundInterceptor } from '@temporalio/interceptors-opentelemetry';

const resource = Resource.default().merge(
  new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'gmail-sync-worker',
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.SERVICE_VERSION,
  })
);

// Configure worker with interceptors
const worker = await Worker.create({
  // ... other config
  interceptors: {
    activity: [
      new OpenTelemetryActivityInboundInterceptor()
    ]
  }
});

// CloudWatch metrics
class MetricsCollector {
  async recordSyncMetrics(userId: string, result: SyncResult) {
    await cloudWatch.putMetricData({
      Namespace: 'GmailSync',
      MetricData: [
        {
          MetricName: 'MessagesAdded',
          Value: result.messagesAdded,
          Dimensions: [{ Name: 'UserId', Value: userId }]
        },
        {
          MetricName: 'SyncErrors',
          Value: result.errors.length,
          Dimensions: [{ Name: 'UserId', Value: userId }]
        }
      ]
    }).promise();
  }
}

1. Security:

   • Store Gmail refresh tokens encrypted

   • Use AWS Secrets Manager for credentials

   • Enable TLS for Temporal connections

   • Implement proper OAuth2 flow

2. Reliability:

   • Set up dead letter queues

   • Implement circuit breakers for Gmail API

   • Configure appropriate retry policies

   • Set up health checks

3. Performance:

   • Batch Gmail API requests

   • Implement message deduplication

   • Use database connection pooling

   • Cache frequently accessed data

4. Cost Monitoring:

   • Set up AWS Cost Explorer alerts

   • Monitor Temporal Cloud usage

   • Track Gmail API quota usage

   • Implement usage-based billing for customers

This architecture provides a robust, scalable, and cost-effective solution for Gmail synchronization using Temporal. The key benefits include:

1. Reliability: Automatic failure handling and retries

2. Scalability: Easy to scale workers based on load

3. Visibility: Built-in workflow tracking and debugging

4. Cost-effective: Pay only for what you use with Temporal Cloud

5. Maintainable: Clear separation of concerns with workflows and activities

For a startup MVP, we can consider starting with Temporal Cloud to minimize operational overhead, then considering self-hosting once we reach ~5,000 active users or need more control over the infrastructure.