[INTERLINKED-22] Broadband exodus

← All Issues || New Issue →

Sort Normal || Priority

Details

Referencing Issues

[INTERLINKED-24] Improve programmatic interfaces to issues

Master issue for broadband exodus from ARTN site

The always-on broadband link at the ARTN site is scheduled to be cut and replaced with intermittent connectivity using the existing POTS line at this site, no later than July 2026.

Initial testing:

• Test dial-up using POTS connection and see if we can reliably get V.90 to an ISP. See if we can borrow any GlobalPOPs/independent accounts. Answer: V.90 doesn't work, but we can consistently get reliable 33.6k connections to GlobalPOPs
• Establish that broadband day-to-day pricing is supported if needed (yes)
• Test if multilink or even concurrent connections (via same account) are possible with GlobalPOPs
• Determine if we will add a second POTS line at this site to support data needs (depends on actual pricing + maximum possible ISP cost compared to broadband connection)
• Test how well RDP works over 31.2/33.6/36k connections
  • Fix ncurses based applications (evergreen) over SSH
  • Fix webmail (wssmail)
• Test 300/1200 bps Softmodem connections to Northern
• Test how well ILBC/G729 (https://github.com/arkadijs/asterisk-g72x) VoIP calls work over 31.2k/33.6k connections (for testing traffic only, not production voice usage) + are there any lower-bandwidth codecs we can use?
  • We may need to create a "pseudo-codec" that doesn't do audio but just passes control frames. This would allow establishing IAX connections that can be used for sending DTMF for testing, etc. Alternately, we could just drop all voice frames so none go over the link and saturate the connection. Audio may require even more compression than ILBC/G729 support potentially
• Test Mosh for SSH connections over dial-up.

Migration (com services):

• Separate DISAs must be established at ARTN (e.g. *) and main sites.
  • The one at the main should be the primary DISA, and a special code should provide authenticated access to the ARTN DISA without requiring reauthentication (e.g. to check messages left there).
  • MWI should be shared between sites, e.g. a new message at ARTN site should trigger MWI at the main, and vice versa. A message/voicemail left on either system should trigger an email, page, etc.
  • Need ability to transfer calls, make 3-way calls, reoriginate, etc.
  • Issue: Access to C*NET 488 doesn't work via the TSPS/DISA, need to investigate and fix
  • Minor Issue: Had to change the out provider from TLS to UDP due to pjsip issues. See if that's fixed with a newer version of Asterisk? Or putz with the configs more.
• IP phones and ATAs with SIP lines connected to WAN will need to cut or migrated to LAN
• Inbound calls. Calls to numbers that would ring at or forward to the site must be configured to alternate route via CCSA over POTS and pass Caller ID + called number forward. We will also need to figure out how to pass answer supervision back . Can pass DNIS+Calling Number using MF.
• Outbound calls. Local PBX will need to use CCSA for call routing. Calls previously sent over IP tie trunk to main must be alternate routed via POTS to the main and send ANI + called number forward.
  • Need ability to transfer calls and make 3-way calls. Also be able to drop current call and make a new one (either directly on remote using * code, or locally, while holding trunk group), for faster setup time / eliminate # of calls
  • Currently, there is an issue in Verizon's network with out of band DTMF detection false positives that has this held up
  • Configure local PBX to make long-distance calls out via CCSA via the POTS line (for 8+, but not 9+), and also be able to send forward the ANI of the POTS line, for calls that would normally be made from the POTS line (this will save on toll charges, and we should even get a B attestation for STIR/SHAKEN instead of a C)
  • A local ARTN code must provide access to voicemail at the main site
  • Alarm reporting line needs to be toll-free accessible so it doesn't use message units.
  • Use A/B/C/D for preemption/priority so POTS line can be seized if needed (e.g. by alarm, for important call, etc.) if voice/data session already active
• Schedule TFC calls (cron)
• Single-ring suppression/delay circuit is needed so the local PBX can answer calls without telephones ringing. Caller ID is not provisioned at this site, so we will use the ring once, hangup, ring again scheme to signal a call for the PBX to answer.
• For push notifications and alerts, we need to be able to receive a call from the main and receive a 1200 baud FSK data transmission that we can process. app_fsk as written doesn't work, using app_softmodem might be the path of least resistance. To make it reliable, we might want to use an ADSI-like scheme that involves checksums/acknowledgments so we can ensure delivery is reliable and corruption-free.
  • Alternately, since email is a good interface for this already, we need the ability from the main to be able to "push" email to the ARTN site. This could either mean somehow establishing a dial-up connection in the reverse direction temporarily, to allow these emails to be downloaded, or using another reliable transport for SMTP in this case besides TCP/IP.
• res_alarmsystem needs to be able to not do reporting for all events, just breaches, as otherwise the phone line would be tied up too much

Migration (net services):

• Download all Wikipedia articles (to supplement encyclopedia) - look into https://en.wikipedia.org/wiki/Kiwix
• Any IP whitelists on the ARTN site need to be eliminated or another method used
• PhreakScript needs to support offline installations (extension to phreaknet source, --offline flag?)
• Cross-site backups using the WAN need to be eliminated (need to save VPS backups outside of the VPS without storing them at ARTN site)
• Split netmon script into ARTN and VPS versions that run independently
• Depending on whether something originates at the main or at ARTN site, the routing logic must be different, i.e. the same destination address should NOT be used on both sides, since that would imply equal routing behavior
  • Any messages sent to pager locally should directly copy the Polycom alerting address and use the TAP application (once written, as in [PHREAKSCRIPT-70]) to dial the pager company and send the page (this way, no Internet connection is needed, and messages can be sent in realtime). Ideally, this would be integrated into the SMTP server to the extent that only when a page is delivered over the POTS line is a 250 OK response sent for DATA. This would allow leveraging native SMTP queuing behavior (with retry logic for these messages only, at least) to ensure reliable delivery of the message if the POTS line is busy, etc.
• Adjust all cron jobs that send daily email jobs to do so during "mail hour" (~1am-4am)
• Reduce the overall number of emails received, esp. unnecessary ones
• Mail server migrations: [INTERLINKED-12] - in particular, emails with large attachments should be rejected with a notice
  • PC1 should be reimaged prior to cutover, after the mail migration (opportune moment) - in particular, with an SSD or RAID-1 of HDDs
  • Any high-importance emails (either marked as high importance, destined for pager, etc.) should be delivered immediately instead of waiting in queue for next queue flush
• Intranet sites with Internet dependencies need to be migrated to the Internet (i.e. grocery price monitor)
  • For the matter, the grocery price monitor should email its output, rather than simply alerting (this way, the entire service could be rehosted on the WAN side, with no caching)
• Provisioning server must be replaced locally (for IP phones and ATAs). This includes the weather display applet. This requires [PHREAKNET-60] to be completed first.
• Caching of WAN services, i.e. calendar download, weather, etc.
  • Weather data for the entire day must be downloaded during mail hour. Services using these must point to the cached version.
• [INTERLINKED-25] - eliminate redundant CSS requests, to reduce bandwidth usage
• HTTP/HTTPS (MITM) caching proxy should be setup, so cached content can be accessed while offline
• Rehost user VMs
• [INTERLINKED-24] - InterLinked Issues - allow creating and commenting on issues by email and also expose issues via IMAP or NNTP
• Rehost the downfiles site, on the WAN side instead.
  • Allow downfiles submissions to be received via SMTP, rather than just by web. This way, we can take advantage of SMTP's offline queuing capabilities.
    • e.g. RECIPIENT downfiles@interlinked.us SUBJECT Download Request BODY https://youtube.com/sdfy87sdf87df
  • Add optimizations to automatically download attachments / linked videos from curated RSS feeds
• Apply for a GETS card
• Use offline TOTP storage service (e.g. locally hosted PHP page)

Unsolved problems:

• 2FA (primarily voice) calls. Can't receive a call while online. Will need to be migrated to a number that can be manipulated out-of-band, i.e. STT email.
• The inverse also applies, e.g. when calling Verizon for change orders, receiving an email for 2FA will no longer work. Migrate to PIN.

Final pre-cutover:

• Forward or disconnect all 2xxx Centrex extensions so they forward to a 3xxx extension which will route in via the POTS line
• Perform measurements at the router of what clients are still accessing the WAN periodically, how much, etc.
• Backup entire media array and replace RAID-0 array with a RAID-1 array using larger disks
  • Emails should be stored on main RAID-10, not the system RAID-1
• Set up local Debian package mirror for frequently used packages
• Download entire Evan Doorbell FLAC collection, any media backlog
• Record a week of Flower Power Radio and set up a local MP3 streaming server that can provide this stream. Also export songs listing locally.
• Sync clocks

Post-cutover, will not be possible:

• Operator working sessions
• ASL repeater operation

Comments

You must be logged in to leave a comment.